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IMPROVING  VISUAL  THREAT  DETECTION:  RESEARCH  TO  VALIDATE  THE  THREAT 
DETECTION  SKILLS  TRAINER 

EXECUTIVE  SUMMARY 


Research  Requirement: 

Visual  threat  detection  remains  an  essential  skill  for  virtually  every  Soldier  in  the 
operational  environment.  The  purpose  of  this  research  was  to  validate  the  Threat  Detection 
Skills  Trainer  (Zimmerman,  Mueller,  Daniels,  &  Vowels,  2012;  Zimmerman,  Mueller,  Grover, 

&  Vowels,  in  preparation),  developed  in  previous  ARI  efforts.  This  trainer  incorporated  the 
findings  from  empirical  research  on  visual  threat  detection,  with  focus  on  threat  search  and 
identification,  attention  management,  and  reasoning  about  threats.  Providing  training  that 
expands  the  perceptual  and  cognitive  skills  associated  with  visual  threat  detection  enhances 
Soldiers’  overall  effectiveness  in  the  operational  environment.  Training  that  enhances  the  ability 
to  recognize  and  understand  relevant  threat  cues  in  operational  settings  allows  Soldiers  to  carry 
out  their  respective  missions  in  a  safer  and  more  effective  manner. 

Procedure: 

To  test  the  effect  of  the  trainer  on  Soldier  skills,  Soldiers  completed  a  series  of  computer- 
based  tasks.  Soldiers  completed  pre-  and  post-tests  that  included  limited  threat  search,  dynamic 
change  tasks,  situational  awareness  tasks,  and  decision-making  exercises.  Soldiers  were 
randomly  assigned  to  a  training  condition,  in  which  they  completed  the  trainer  between  the  pre- 
and  post-test,  or  a  control  or  no  training  condition,  in  which  they  completed  a  filler  task. 
Debriefing  occurred  after  the  experiment  alongside  a  brief  interview  regarding  realism  of  the 
training  tasks.  All  Soldiers  assigned  to  the  control  condition  had  the  opportunity  to  complete  the 
trainer  later,  if  they  chose. 

Findings: 

Soldiers  reported  finding  the  trainer  valuable,  despite  no  effect  on  threat  detection 
perfonnance  between  the  training  and  no-training  conditions.  Results  from  all  four  tasks, 
quantitative  and  qualitative,  indicated  improvement  over  time.  This  finding  may  have  resulted 
from  simple  practice  effects.  These  effects  appeared  to  be  consistent  and  robust,  with  effect 
sizes  that  were  sometimes  relatively  large.  Across  six  statistically  significant  findings,  partial 
eta~  ranged  from  0.146  to  0.367  (M=  0.25,  SD  =  0.08).  This  suggests  that,  on  average,  practice 
effects  (or  some  other  effect  of  the  pre-post  manipulation)  accounted  for  25%  of  increase  in 
perfonnance  over  time.  The  limitation  of  significant  findings  from  the  primary  (training) 
manipulation  may  stem  from  low  statistical  power  resulting  in  limited  ability  to  detect  subtle 
differences  between  conditions.  The  overall  positive  outcome  is  that  all  participants  improved 
on  all  of  the  primary  skills  of  visual  threat  detection  with  only  a  1 .5  to  2  hour  training  session. 
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Utilization  and  Dissemination  of  Findings: 

The  findings  from  this  research,  while  insufficient  to  support  and  validate  the  trainer  as 
an  effective  method  of  training  or  enhancing  visual  threat  detection  ability  in  Soldiers,  do  not 
suggest  that  the  trainer  is  ineffective.  The  tasks  in  the  pre-  and  post-tests  were  similar  to  those 
presented  in  the  trainer.  Across  both  conditions,  improvement  occurred,  suggesting  that  these 
tasks  did  have  a  positive  impact  on  performance.  Soldiers  with  varying  levels  of  experience 
strongly  expressed  that  the  tool  was  interesting  and  challenging,  potentially  serving  as  a  novel 
aid  to  existing  training.  Further  research  should  achieve  higher  statistical  power  to  better 
measure  the  validity  and  effectiveness  of  the  threat  detection  skills  trainer.  The  improvement  in 
perfonnance  scores  would  have  to  be  investigated  over  a  longer  time  to  detennine  if  such  acute, 
cognitive  interventions  are  effective.  Further,  such  a  longitudinal  finding  could  be  very 
beneficial  for  operational  units  who  have  a  large  amount  of  training  to  accomplish  in  a  limited 
training  calendar. 
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IMPROVING  VISUAL  THREAT  DETECTION:  RESEARCH  TO 
VALIDATE  THE  THREAT  DETECTION  SKILLS  TRAINER 

Introduction 

Previous  research  identifying  the  perceptual  and  cognitive  skills  of  Soldiers  during  visual 
threat  detection  tasks  (Zimmerman,  Mueller,  Daniels,  &  Vowels,  2012;  Zimmerman,  Mueller, 
Grover,  &  Vowels,  in  preparation)  resulted  in  development  of  a  training  exemplar,  the  “Threat 
Detection  Skills  Trainer,”  also  referred  to  as  the  TDST  for  the  remainder  of  the  document.  The 
intent  of  the  trainer  is  to  improve  Soldier  threat  detection  in  dynamic  environments.  Previous 
research  enabled  us  to  identify  the  relevant  threat  decision  skills  for  the  trainer  (see  the  model  of 
threat  detection;  Zimmerman  et  ah,  in  preparation).  Even  so,  it  was  unclear  how  using  the  trainer 
may  actually  enhance  Soldiers’  threat  detection  skills.  Thus,  the  purpose  of  the  current  research 
was  to  determine  the  effectiveness  of  this  trainer  and  identify  how  the  U.S.  Anny  can  use  the 
trainer  to  support  their  need  for  efficient  and  effective  training  of  visual  threat  detection. 

Threat  Detection  Skills  Trainer  (TDST) 

The  TDST  was  designed  to  improve  Soldiers’  ability  to  (a)  identify  threats,  (b)  manage 
attention,  and  (c)  engage  in  causal  reasoning  about  threats.  The  summative  effect  of  this  training 
is  to  develop  Soldiers’  mental  models  through  experiential  learning  in  operationally  relevant 
computerized  tasks  and  to  improve  threat  reasoning  ability.  As  a  result,  Soldiers  will  be  better 
able  to  recognize  threats  and  react  in  threat  situations.  To  accomplish  this,  the  trainer 
incorporates  three  sets  of  exercises  requiring  Soldiers  to  interact  with  digital  photos  depicting 
operational  environments  where  potential  threats  may  exist.  In  the  first  set  of  exercises,  Soldiers 
view  photos  marked  and  annotated  with  threat  cues  previously  identified  by  experienced 
Soldiers.  Soldiers  click  on  marked  threat  cues  to  read  about  the  threat,  its  relevance,  and  the 
common  enemy  tactics  associated  with  that  threat.  Soldiers  then  view  photos  without  pre¬ 
marked  threat  cues  and  click  on  potential  cues.  If  the  cues  are  relevant,  an  annotation  appears 
that  provides  information  about  that  cue.  After  the  threat  search  task,  Soldiers  complete  a  timed 
threat  search  exercise,  in  which  they  have  limited  time  to  identify  as  many  relevant  threat  cues  as 
possible.  This  first  set  of  exercises  enhances  ability  to  identify  threats  by  demonstrating  what 
threats  look  like,  where  they  are  likely  to  occur,  and  the  reasons  they  occur. 

In  the  next  set  of  exercises,  Soldiers  view  photos  and  answer  questions  about  the  content 
depicted  in  the  photos.  The  questions  challenge  Soldiers  to  think  critically  about  the  scene’s 
context,  consider  potential  threats,  and  generate  possible  solutions  for  addressing  the  threats. 
During  these  exercises,  Soldiers  review  statements  made  in  previous  research  by  experienced 
Soldiers  who  evaluated  the  threats  in  the  photos. 

Following  this  exercise,  Soldiers  review  written  scenarios  that  provide  background  and 
context  to  select  photos.  They  read  the  scenarios  and  answer  questions  about  potential  threats 
and  the  relevance  of  threats  in  the  photos.  These  exercises  enhance  Soldier  ability  to  engage  in 
causal  reasoning  about  threats  by  requiring  them  to  think  about  the  greater  context  in  which 
potential  threats  exist  and  to  identify  threats  based  not  only  on  physical  features  of  the 
environment,  but  also  on  other  contextual  features. 
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At  two  points  during  the  previous  exercises,  Soldiers  engage  in  two  change  detection 
tasks.  In  those  exercises,  Soldiers  view  edited  versions  of  photos  that  they  saw  during  a  previous 
exercise.  The  edited  photos  contain  added  or  removed  threat-relevant  and  threat-irrelevant  items 
and  Soldiers  attempt  to  identify  the  changes.  This  exercise  was  designed  to  enhance  Soldiers’ 
ability  to  manage  attention  by  forcing  them  to  attend  to  threat-relevant  areas  of  environments  and 
detect  relevant  changes. 

Model  of  Threat  Detection 

Vowels  (2010)  has  argued,  to  detect,  evaluate,  and  defeat  threats  in  an  operational 
environment,  Soldiers  must  perceive  visual  information  and  comprehend  that  information  within 
the  context  of  the  operational  environment.  Only  by  doing  this  effectively  are  they  able  to  make 
accurate  decisions  about  potential  threats  in  an  operational  environment.  To  comprehend  an 
environment,  Soldiers  must  effectively  search  for,  identify,  and  encode  threat  relevant 
information.  One  outcome  of  previous  research  was  a  model  of  threat  detection  that  identified  a 
set  of  measurable  behaviors  experienced  decision  makers  would  likely  exhibit  during  threat 
detection  tasks  (Zimmerman,  et  ah,  2012;  Zimmerman,  Mueller,  Marcon,  Daniels,  &  Vowels, 
2011).  The  model  depicts  a  cycle  of  threat  detection  that  includes  (a)  monitoring  and  search 
activities  required  to  maintain  situation  awareness  when  no  overt  threats  are  present,  (b) 
identifying  and  evaluating  potential  threats,  and  c)  concluding  a  threat  is  present  and  determining 
a  course  of  action  (Figure  1).  Associated  with  the  model  are  behaviors  that  experienced  threat 
detectors  likely  engage  in  as  they  attempt  to  detect  threats. 


Figure  1.  Threat  Detection  Loop  (inner  rectangles)  and  measurable  expert  threat  detection  behaviors 
(outer  rectangles).  From  Detennining  the  Requisite  Components  of  Visual  Threat  Detection  to 
Improve  Operational  Performance  (ARI  Technical  Report)  by  Zimmerman,  L.  A.,  Mueller,  S.  T., 
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Grover,  J.,  &  Vowels,  C.  L.  (in  preparation).  Fort  Belvoir,  VA:  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences. 

The  TDST  trainer  engages  Soldiers  in  tasks  where  they  monitor  environments,  engage  in 
search  activities,  and  identify  potential  threats.  In  particular,  this  training  is  aimed  at  enhancing 
Soldier  ability  to  encode  details  about  a  situation  (Refer  to  Figure  1):  (a)  encode  situational 
details,  (b)  classify  threat  versus  non-threat  features,  (d)  incorporate  prior  information  into 
decisions,  (g)  detect  changes  or  anomalies,  (h)  create  causal  stories  about  threat  situations,  and 
(i)  conduct  efficient  visual  search  based  on  knowledge.  The  training  encompasses  four  learning 
objectives  (Table  1).  These  learning  objectives  map  onto  the  expected  behaviors  identified  as 
part  of  the  threat  detection  loop  (Figure  1).  For  instance,  the  initial  exercises  focus  on  encoding 
details  to  identify  and  classify  relevant  threats.  The  goal  of  the  reasoning  tasks  is  to  increase 
Soldier  ability  to  gather  evidence  to  determine  threat  importance  and  impact.  The  change 
detection  tasks  focus  on  perceiving  changes  and  determining  the  relevance  of  the  change. 

Table  1 

Relationship  between  Learning  Objectives  and  the  Threat  Detection  Loop 


Learning  Objectives 

Step  in  Threat  Threat  Detection 

Detection  Loop  Loop  Behaviors 

1 .  Demonstrate  an  improved 
ability  to  identify  relevant 
threats  in  a  variety  of 
situations. 

•  Monitoring,  vigilance,  and  a)  Encoding  details  for  immediate 
search  activities  or  delayed  test 

2.  Distinguish  between  threat¬ 
relevant  and  threat-irrelevant 
cues  in  time -limited  situations. 

•  Identify/classify  threat  or  b)  Ability  to  classify  threat  vs. 

non-threat  non-threat  features 

i)  Ability  to  look  at  high-priority 
threat  locations  based  on 
knowledge 

3.  Recognize  the  importance  and 
potential  impact  of  each 
detected  threat. 

•  Identify/classify  threat  or  c)  Ability  to  adapt  decision 

non-threat  selectivity  based  on 

•  Gather  evidence  to  ID  information 

threat  f)  Ability  to  balance  decision  with 

information  gathering 
h)  Ability  to  create  causal  stories 
about  threat  situations 

4.  Identify  threat-relevant 
changes  in  the  environment. 

•  Change  or  anomaly  g)  Ability  to  detect  change  or 

detection  anomaly 

It  is  possible  to  objectively  identify  changes  in  performance  resulting  from  training, 
because  these  behaviors  can  be  assessed  empirically.  To  determine  if  the  TDST  meets  the 
learning  objectives  and  the  overall  goal  of  improving  Soldier  ability  to  detect  and  identify 
relevant  threats,  we  conducted  an  experiment  by  measuring  inexperienced  Soldier  performance 
before  and  after  completing  the  training.  The  performance  of  the  experimental  group  was 
compared  to  a  control  group  of  Soldiers  who  had  completed  the  same  before  and  after  exercises 
without  the  training.  During  the  experiment,  Soldiers  engaged  in  four  tasks,  each  intended  to 
measure  threat  detection  behaviors.  Table  2  lists  the  tasks  and  the  behaviors  measured  in  each 
task. 
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Table  2 


Experimental  Tasks  and  Associated  Behaviors 


Task 

Threat  Detection  Behavior 

Limited  threat  search 

• 

• 

Look  at  high-priority  threats  based  on  knowledge  (i) 

Classify  threat  versus  non-threat  features  (b) 

• 

Look  at  high-priority  threats  based  on  knowledge  (i) 

Dynamic  change  task 

• 

• 

Classify  threat  versus  non-threat  features  (b) 

Detect  change  or  anomaly  (g) 

• 

Encode  details  for  immediate  or  delayed  test  (a) 

Situation  awareness  task 

• 

• 

Look  at  high-priority  threats  based  on  knowledge  (i) 

Create  causal  stories  about  threat  situations  (b) 

• 

Classify  threat  versus  non-threat  features  (b) 

Decision-making  exercise 

• 

• 

Create  causal  stories  about  threat  situations  (h) 

Incorporate  prior  knowledge  into  decisions  (d) 

Experimental  Tasks 

Limited  threat  search. 

In  previous  research  (Zimmerman,  Mueller,  Daniels,  &  Vowels,  2012;  Zimmerman, 
Mueller,  Grover,  &  Vowels,  in  preparation),  Soldiers  identified  and  annotated  potential  threats  in 
a  set  of  static  images.  After  reading  a  brief  background  story  for  each  image,  Soldiers  viewed 
each  image  and  clicked  with  a  computer  mouse  pointer  on  areas  and  items  they  judged  to  be  the 
greatest  threat.  They  rated  the  likelihood  and  severity  of  an  actual  threat  occurring  in  that 
location  on  a  3-point  scale  (low,  medium,  high).  After  providing  ratings,  Soldiers  annotated  the 
location  with  a  short  description  of  the  potential  threat  (see  Figure  2).  Soldiers  provided  between 
three  and  1  points  of  interest  per  image. 


I  monuments  require  careful  movement  as  to  not  disrespect  locals 

2 1  would  be  weary  of  a  vacant  area,  the  local  population  may  be  aware  of  an  upcoming  attack 

3  enemy  attack  position/sniper 

4  walls  provide  loc.  for  lEDs 

5  pot  for 'booby  traps' 

6  enemy  small  arms/RPG  attack  positions 

7  Snllperon  hilltop 

8  Good  Sniper  location 

9  possible  threat 

10  Possible  mortar  spotters 

I I  Snipers  or  RPG  teams 

1 2  lEDs  attached  to  inside  of  wall 

1 3  harassing  smallarms-  choke  point 


1 4  traffic  circles  slow  traffic  down  making  it  easier  to  hit  a  target 

1 5  ambush  small  arms 

1 6  rockets, morters 

1 7  insergent  hideout  possible 

18  Sniper  postion 

19  High  ground  is  good  forobservers.snipers 

20  Looks  like  an  open  garage  door, sniper  could  use  darkness  as  concealment 

21  Empty  streets  could  Indicate  Inhabitants  know  something  is  about  to  happen 

22  Dark  areas  could  conceal  a  sniper 

23  This  high  point  offers  the  enemy  great  observation  with  an  easy  escape  and  good  cover. 

24  Abandonded  monuments  are  likely  IED  locations 

25  Abandoned  buildings  offer  the  enemy  many  places  to  hide  and  cache  weapons, ammunitions, and  materials. 

26  No  locals  can  indicate  some  thing  bad  is  about  to  happen 

27  These  bushes  could  be  a  good  place  to  put  an  IED 

28  Good  place  for  a  sniper  or  ambush 

29  patrol  bldg  for  possiable  inhabitants, caution  for  any  explosives 

30  if  monument  Is  unlnhabltated  very  likly  tobe  attacked  for  inside  or  around  it 

31  small  arms  fire  or  vbied 

32  shady  areas  provide  good  concealment  for  insurgents 

33  rpg  or  command  detnators 

34  Direct  attack  here  initiated  to  draw  US  forces  to  the  site  creating  bad  news  report 

35  sniper  location 

36  sniper  location 

37  possible  IED  in  turn  around,  there  are  no  cars  here  but  plent  in  the  rear  right  of  the  picture 

38  there  Is  no  one  around 

39  elevated  position 

40  possible  enemy  position 

41  big  truck, could  be  carrying  weapons.or  explosives 

42 

43  open  area.shaded,  for  sniper  activity 

44  ambush  sites  -high  ground 

45  ambush  site 

46  Two  Taxis  means  someone  Is  there 

47  Shooters  and  rubble  go  together 

48  New  curbing  and  rock, built  In  IED 

49  Sniper/lookout 

50  Rubble, hard  to  detect  threats 

51  Large  mass  vehicle 

52  Black  flags  could  be  warning  of  a  possible  attack. 

53  Observation  from  the  enemy  on  the  high  ground. 

54  Enemy  observation  and  triggerman  location  in  the  buildings. 


Figure  2.  Example  of  image  threat  search  and  annotations. 
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We  applied  this  measurement  method  during  a  previous  experiment  (Zimmerman  et  al., 
in  preparation).  During  that  task,  Soldiers  searched  an  image  and  clicked  on  locations  where 
potential  threat  targets  existed.  For  portions  of  the  task,  Soldiers  had  limited  resources  (a  limit  of 
10  clicks)  and  limited  time  and  resources  (a  maximum  of  one  click  per  second  for  10  seconds). 
Findings  from  that  previous  research  indicated  Soldiers  with  deployment  experience  performed 
slightly  better  when  under  time  pressure,  whereas  time  pressure  had  greater  negative  influence 
on  Soldiers  with  no  experience. 

In  the  current  research,  these  images  provided  a  measure  of  performance  on  a  threat 
search  task.  Soldiers  clicked  on  locations  in  images  where  threats  might  exist,  using  a  limited 
number  of  clicks.  The  closer  their  clicks  were  to  the  locations  where  experienced  Soldiers 
clicked  in  the  previous  experiment,  the  more  closely  we  can  say  their  performance  matches 
someone  with  experience  (proficiency).  We  hypothesized  that  the  clicks  would  be  closer  post¬ 
training  compared  to  pre-training. 

Dynamic  Change  Task. 

Soldiers  in  the  current  research  attempted  to  respond  to  threats  that  appeared  in 
operationally  relevant  photos.  In  the  previous  research  (Zimmerman,  et  al.,  2012)  Soldiers 
completed  a  dynamic  threat  detection  task  by  guiding  their  visual  attention  over  images  obscured 
by  vibrating  transparent  dots.  Soldiers  were  required  to  maintain  constant  vigilance  as  they 
attempted  to  detect  a  target  dot  that  appeared  briefly  at  uneven  intervals  during  the  viewing.  The 
target  dot  would  appear  at  threat  relevant  and  threat  irrelevant  locations.  Soldiers  clicked  a 
computer  mouse  button  when  the  target  was  detected,  after  which  a  cursor  appeared  and  Soldiers 
indicated  the  location  of  the  target.  The  current  research  deviated  from  this  by  incorporating 
change  detection  with  threat  detection  and  by  updating  the  stimuli  to  include  threat  relevant  and 
threat  irrelevant  targets  (rather  than  semi-transparent  dots). 

In  previous  research  (Zimmerman,  et  al.,  2012),  Soldiers  engaged  in  a  change  detection 
task  by  identifying  changes  in  previously  viewed  photos.  Findings  revealed  high  false  alarm 
rates;  however,  when  Soldiers  did  identify  actual  changes,  the  majority  of  these  were  in  threat¬ 
relevant  locations.  In  addition,  Solders  were  more  likely  to  identify  the  changes  when  the  second 
photo  contained  an  object  that  was  absent  from  the  first  photo  compared  to  when  the  object  was 
removed  in  the  second  photo. 

The  goal  of  the  dynamic  change  task  was  to  determine  if  Soldiers  focus  attention  on 
threat-relevant  locations  to  detect  changes  that  occur  in  those  locations.  We  hypothesized  that 
Soldiers  would  detect  more  changes  in  threat-relevant  locations  and  miss  more  changes  in  threat 
irrelevant  locations  after  training.  We  also  predicted  that  reaction  time  would  be  quicker  when 
identifying  threats  in  relevant  locations.  This  task  utilized  the  Flicker  paradigm  (Rensink, 
O’Regan,  &  Clark,  1997;  Scholl,  2000).  Soldiers  viewed  each  photo  multiple  times  and  in  some 
viewings,  a  single  object  in  the  photo  changed.  The  photos  flashed  on  the  screen  and  between 
flashes,  a  mask  concealed  the  photo.  Soldiers  viewed  the  changed  and  unchanged  photos  until 
they  identified  the  location  of  the  change. 
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Situation  Awareness  Task. 


The  purpose  of  the  situation  awareness  task  was  to  detennine  if  trained  Soldiers  are  able 
to  perceive  and  comprehend  elements  in  an  environment  more  effectively  than  those  who  have 
not  completed  the  TDST.  The  ability  to  perceive  and  comprehend  information  constitutes  the 
first  two  levels  of  situation  awareness  (Endsley,  2000).  According  to  Endsley,  situation 
awareness  occurs  at  three  levels.  Level  1  is  the  ability  to  perceive  objects  in  an  environment, 
Level  2  is  the  ability  to  comprehend  their  meaning,  and  Level  3  is  the  ability  to  project  their 
status  into  the  future.  To  test  both  Level  1  and  2  situation  awareness,  we  used  video  recordings 
of  Soldier  activities  in  Afghanistan  retrieved  from  military  websites.  These  videos  depict  various 
environments,  ranging  from  dismounted  patrols  to  convoy  operations  that  offer  relevant 
environmental  characteristics.  For  example,  the  dismounted  patrol  environment  offers  building 
structures  (both  pennanent  and  temporary),  individual  persons  and  crowds,  cars,  potential 
surveillance  or  hiding  points,  and  potentially  suspicious  behavior.  Being  able  to  monitor  and 
search  an  environment  and  subsequently  encode  details  about  the  situation  are  precursors  to 
detecting  relevant  threats. 

Level  1:  Perception.  To  test  perception,  Soldiers  viewed  short  video  segments  that 
stopped  abruptly.  At  this  point,  a  wireframe  drawing  of  the  last  frame  in  the  video  appeared  and 
Soldiers  placed  cutouts  of  items  (e.g.,  people,  vehicles,  rocks,  windows)  from  the  video  into  the 
wireframe  at  the  approximate  location  of  those  items  (see  Figure  3).  The  items  in  the  scenes 
represented  cues  that,  based  on  previous  research  findings,  were  relevant  to  identifying  and 
comprehending  threats.  In  addition  to  the  items  from  the  video  (targets),  Soldiers  could  select 
items  that  were  not  in  the  video  (foils).  Each  exercise  included  target  items  and  two  foil  items. 
Analysis  included  accuracy  in  item  choosing  and  location. 


Figure  3. 
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Level  2:  Comprehension.  After  Soldiers  identified  items  in  these  images,  they  described 
the  potential  meaning,  relevance,  and  challenges  associated  with  these  cues  (Appendix  A).  We 
evaluated  these  descriptions  for  level  of  detail  and  comprehension.  We  also  coded  comments 
that  were  characteristic  of  experienced  Soldiers,  such  as  hypotheses  about  what  is  occurring  in 
the  scenes,  enemy  strategies  and  motivations,  and  strategies  they  would  use  to  deal  with  the 
situation. 

We  hypothesized  that  Soldiers  who  completed  the  TDST  would  select  threat-relevant 
items  (targets)  more  often  than  threat-irrelevant  items  (foils)  and  they  would  be  more  accurate  in 
placing  target  items  in  correct  locations  compared  to  threat-irrelevant  locations.  They  would  also 
provide  descriptions  that  include  hypotheses  about  the  situation  and  strategies  for  dealing  with 
the  threats.  We  also  expected  that  Soldiers  who  completed  the  training  would  provide 
descriptions  that  indicated  they  comprehend  the  meaning  of  the  items  in  the  scene,  particularly 
threat-relevant  items. 

Decision-Making  Exercises  (DMX). 

The  purpose  of  the  DMXs  was  to  detennine  if  training  improved  Soldier  ability  to  create 
causal  stories  about  threat  situations  and  incorporate  prior  information  into  their  decisions. 
Previous  findings  (Zimmerman,  Mueller,  Daniels,  &  Vowels,  2012;  Zimmerman,  Mueller, 
Grover,  &  Vowels,  in  preparation),  showed  that  Soldiers  with  more  experience  (as  quantified  by 
the  number  of  deployments)  provided  complex  narratives  about  detecting  threats  that  included 
the  strategies,  skills,  and  challenges  associated  with  detecting  threats.  While  experienced 
Soldiers  constructed  rich  accounts  of  threat  detection  situations,  novice  Soldiers  (with  no 
deployment  experience)  often  gave  context-free  assessments  of  the  situations,  providing 
responses  that  indicated  they  were  aware  of  important  cues  that  indicated  threats,  but  lacked  the 
ability  to  incorporate  these  cues  into  meaningful  narratives. 

We  relied  on  findings  from  interviews  conducted  in  previous  research  (Zimmerman,  et 
ah,  2012;  Zimmerman,  et  ah,  in  preparation)  and  our  military  subject  matter  expert  to  create 
engaging  and  realistic  scenarios  that  reflected  current  threat  detection  contexts.  The  scenarios 
included  elements  of  uncertainty,  relevant  and  irrelevant  cues,  and  conflicting  goals  (Klein, 
2004).  We  hypothesized  that  Solders  who  completed  the  TDST  would  provide  descriptions  that 
demonstrate  more  advanced  decision  characteristics.  The  Dreyfus  and  Dreyfus  model  of  skill 
acquisition  provided  metrics  for  classifying  situation  assessment  and  decisions  on  a  range  from 
‘novice’  to  ‘expert’  (Dreyfus  &  Dreyfus,  1986;  Ross,  Phillips,  Klein,  &  Cohn,  2005). 
Assessments  of  skill  level  took  into  account  several  factors,  such  as  the  strategies,  cues,  and 
challenges  that  Soldiers  discussed. 

The  goal  of  the  DMX  task  was  to  identify  any  changes  in  Soldier  causal  reasoning  after 
completing  the  TDST.  During  the  DMXs,  Soldiers  read  a  scenario  and  viewed  a  photo  that 
represented  their  current  view  of  the  situation.  After  they  read  the  scenario,  they  had  two 
minutes  to  provide  a  solution  to  the  problem  presented  in  the  scenario  (Appendix  B).  After 
completing  their  narrative,  Soldiers  answered  several  questions  aimed  at  assessing  their  critical 
thinking  and  causal  reasoning  skills. 
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Summary 


The  research  objective  for  this  project  was  to  determine  how  the  TDST  could  impact 
threat  detection  skills.  We  hypothesized  that  Soldiers  who  completed  the  threat  detection  skills 
training  would  perform  better  on  the  four  identified  tasks  compared  to  Soldiers  who  did  not 
complete  the  training.  We  also  expected  to  see  improved  perfonnance  after  training  compared  to 
before  training.  Specifically,  we  predicted  that  trained  Soldiers  would: 

•  Identify  relevant  threats  faster  and  more  accurately  by  clicking  on  threat  locations  that 
reflect  the  locations  previously  chosen  by  experts; 

•  Detect  more  changes  in  threat  relevant  locations  and  miss  more  changes  in  threat 
irrelevant  locations; 

•  Recall  more  details  when  asked  to  describe  a  current  situation  from  memory  and 
provide  explanations  about  elements  of  the  situation; 

•  Identify  and  be  more  accurate  in  placing  threat  relevant  items  (targets)  in  correct 
locations  compared  to  threat  irrelevant  items  (foils); 

•  Provide  descriptions  of  the  scene  that  indicated  they  comprehend  the  meaning  of  the 
items  in  the  scene,  particularly  threat-relevant  items;  and, 

•  Provide  assessments  of  situations  that  include  elements  indicative  of  more  expert 
decision  skills. 

Method 

Participants, 

Forty-three  U.S.  Army  Soldiers  participated  in  this  research.  All  Soldiers  were  male. 

The  mean  age  of  participants  was  21  years  (range  19-37).  Five  Soldiers  deployed  once,  and  one 
Soldier  deployed  twice  (14%  total).  Of  those  Soldiers  who  reported  previous  deployment,  three 
indicated  going  outside  the  wire  more  than  once  a  month,  two  reported  going  out  once  a  month, 
and  one  reported  going  outside  less  than  once  a  month.  The  average  time  in  service  across  all 
Soldiers  was  17  months  (range  9-60  months).  The  majority  of  Soldiers  had  a  rank  of  private  first 
class  (PFC)  and  ranged  from  private  (PVT)  to  Corporal/Specialist  (CPL/SPC).  Table  3  lists  the 
number  of  Soldiers  in  each  rank,  the  number  deployed  in  each  rank,  and  the  MOS  by  rank.  The 
five  Soldiers  with  MOSs  other  than  1  IB  had  deployed,  as  did  one  Soldier  with  an  MOS  of 
infantryman  ( 1  IB). 
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Table  3 


Soldier  Rank,  Number  Deployed,  and  MOS 


MOS 


Rank 

Total 

Deployed 

11B 

19K 

91B 

91M 

Private 

2 

0 

2 

PV2 

1 

0 

1 

PFC 

24 

1 

23 

1 

CPL/SPC 

16 

5 

12 

1 

1 

2 

Note:  MOS  are  1  IB  =  Infantryman,  19K  =  Ml  Armor  Crewman,  9 IB  =  Wheeled  Vehicle 
Mechanic,  91M  =  Bradley  Fighting  Vehicle  System  Maintainer. 

Design. 

This  research  was  a  2  (training:  training  vs.  no  training)  x  2  (pre/post:  pre-test  vs.  post¬ 
test)  mixed  design,  with  training  being  a  between-subjects  factor.  We  randomly  assigned 
Soldiers  to  the  training  condition.  Twenty-two  Soldiers  completed  the  training. 

Materials. 

Soldiers  completed  all  tasks  and  the  threat  detection  training  on  laptop  computers.  They 
viewed  a  variety  of  photos  and  videos  depicting  relevant  military  situations,  read  content,  and 
made  responses  by  using  a  computer  mouse  to  click  on  the  computer  screen.  They  also  provided 
descriptions  and  answered  questions  using  the  keyboard.  Thus,  we  had  written  entries  for  all  of 
the  questions  to  which  Soldiers  responded. 

Computer-controlled  stimuli  presentation.  We  used  the  Psychology  Experiment 
Building  Language  (PEBL)  0.12  computer  experimentation  system  to  control  the  computer 
exercises  (Mueller,  2009).  All  stimuli  and  questions  were  programmed  into  PEBL  and  Soldiers 
completed  the  experiment  by  viewing  and  responding  in  this  program.  The  PEBL  recorded  all 
responses  for  later  analysis. 

Military  photographs.  We  chose  photos  for  this  experiment  from  among  a  set  of  photos 
retrieved  during  research  conducted  to  develop  the  TDST  (Zimmerman,  et  ah,  2012).  These 
images  were  retrieved  from  three  main  sources,  www.DefenseImagery.mil, www.Flickr.com, 
and  www.defense.gov.  All  images  were  approved  for  public  release.  These  photos  appeared  in 
the  limited  threat  search  task,  dynamic  change  detection  task,  and  decision-making  exercises. 

Military  videos.  The  U.S.  Army  Research  Institute  retrieved  a  random  selection  of 
military  videos  depicting  recent  operations  in  Afghanistan  and  Iraq  from  the  Defense  Imagery 
Management  Operations  Center  (DIMOC).  All  videos  were  approved  for  public  release.  We 
reviewed  these  videos  and  selected  relevant  scenes  depicting  threat  detection  challenges.  The 
average  length  of  the  edited  video  clips  was  12  seconds  (range  7s  to  13s). 


9 


EOF  video  stills.  In  addition  to  the  military  videos  depicting  live  events,  we  also  used 
still  snapshots  of  simulated  videos  depicting  challenging  decision  points  in  military  situations. 
The  U.S.  Army  Research  Institute  recently  produced  a  series  of  Virtual  Battle  Space  2  (VBS2) 
escalation  of  force  (EOF)  exercises  (Johnston,  Leibrecht,  Topolski,  Vowels,  &  Singer,  in 
preparation).  Soldiers  viewed  snapshots  of  these  exercises  to  complete  two  decision-making 
exercises.  Figure  4  shows  a  screen  shot  of  an  EOF  exercise. 


Figure  4.  VBS2  EOF  video  still  for  use  in  the  decision-making  exercise. 

Filler  task.  Soldiers  who  did  not  complete  the  training  engaged  in  a  filler  task  for 
approximately  30  minutes.  The  filler  task  included  games  such  as  word  find,  Sudoku,  and 
crossword  puzzles.  Soldiers  could  choose  which  games  they  wanted  to  play.  Soldiers  completed 
the  paper-based  games  using  pen  or  pencil.  They  could  take  the  games  with  them  when  the 
experiment  was  complete. 

Computer-controlled  questionnaire.  Soldiers  completed  a  demographic  questionnaire  as 
a  part  of  the  computerized  tasks.  The  questionnaire  included  demographic  questions  along  with 
questions  about  education  and  military  experience  (Appendix  C). 

Procedure. 

This  experiment  lasted  approximately  two  hours  per  Soldier.  The  first  five  minutes  of  the 
experiment,  involved  briefing  Soldiers  on  the  nature  of  the  tasks  and  then  collecting  informed 
consent.  Soldiers  then  completed  a  practice  exercise  to  ensure  they  understood  the  tasks. 
Following  this,  they  participated  in  the  pre-test  by  completing  the  four  tasks.  After  a  5-minute 
break,  the  Soldiers  in  the  training  condition  completed  the  threat  detection  skills  training  while 
Soldiers  in  the  no  training  condition  engaged  in  the  filler  task.  The  training  took  approximately 
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30-35  minutes  to  complete.  After  another  5-minute  break,  all  Soldiers  completed  the  post-test 
(see  Table  4).  The  four  tasks  included  in  the  pre-  and  post-tests  were  counterbalanced  to  control 
for  order  and  sequence  effects.  The  stimuli  in  each  task  (e.g.,  photos)  were  also  counterbalanced 
when  applicable.  Following  the  post-test,  Soldiers  completed  a  brief  demographic  questionnaire 
and  provided  any  final  comments.  Finally,  a  researcher  debriefed  the  Soldiers  individually  or  in 
groups  as  they  finished  the  experiment.  Soldiers  had  the  opportunity  to  ask  questions  during  the 
debriefing  session. 

Table  4 

Approximate  Time  to  Complete  Each  Task 


Task 

Time  to  complete 

Informed  Consent  and  Briefing 

5  minutes 

Pre-test  (including  practice  exercise) 

30-35  minutes 

Break 

5  minutes 

Training  or  Filler  Task 

30-35  minutes 

Break 

5  minutes 

Post-Test  (including  demographics  and  debrief) 

30-35  minutes 

Limited  Threat  Search.  Soldiers  monitored  static  scenes  and  identified  possible  threats 
by  clicking  on  items  in  the  scenes.  They  were  limited  to  a  maximum  of  10  clicks  and  15  seconds 
per  photo.  The  limited  resources  (i.e.,  clicks  and  time)  required  them  to  adopt  a  more 
conservative  selection  criterion  while  the  limited  time  added  time  pressure.  The  purpose  of  these 
constraints  was  to  force  Soldiers  to  focus  their  attention  on  threat  relevant  locations  and  choose 
only  high-priority  threats.  Soldiers  completed  this  task  by  viewing  a  series  of  photos  and 
clicking  with  a  computer  mouse  on  the  photos  in  threat  relevant  locations  that  they  deemed  a 
high-priority.  We  evaluated  Soldier  performance  by  comparing  their  responses  to  the  responses 
of  experienced  Soldiers  in  previous  research. 

Dependent  variables  recorded  in  this  task  included  the  locations  chosen,  weighted  by  the 
order  in  which  they  were  selected,  and  the  time  profile  of  the  searches.  Independent  variables 
included  threat  relevance  and  threat  location  based  on  previous  research  data. 

Dynamic  Change  Detection.  Soldiers  viewed  the  changed  and  unchanged  photos  until 
they  identified  the  location  of  the  change  or  until  20  seconds  elapsed,  at  which  time  they  were 
required  to  identify  a  location  where  they  thought  a  change  had  occurred.  Each  altered  and 
unaltered  photo  appeared  several  times  in  a  row.  Once  Soldiers  detected  the  change,  they 
indicated  where  they  saw  the  change  by  clicking  that  area  on  the  photo.  Once  they  clicked  their 
chosen  spot,  the  location  of  the  actual  change  appeared.  The  purpose  of  task  was  to  determine  if 
Soldiers  noticed  changes  in  threat  relevant  areas  faster  and  more  often  than  changes  in  threat 
irrelevant  areas. 

Dependent  variables  recorded  in  this  task  included  change  detection  accuracy,  frequency 
of  detecting  threats  in  relevant  versus  irrelevant  locations,  and  response  latency.  Independent 
variables  included  manipulation  of  change  locations  based  on  relevancy. 
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Situation  Awareness .  Soldiers  viewed  videos  of  U.S.  Soldiers  conducting  vehicle  and 
foot  patrols  in  Afghanistan.  The  videos  stopped  abruptly  during  the  action  and  a  wireframe  of 
the  last  frame  of  the  video  appeared  alongside  several  items.  Each  video  lasted  an  average  of  1 1 
seconds  (range  7  to  12s).  Soldiers  then  placed  the  items  from  the  video  into  the  wireframe  at  the 
location  they  remembered  seeing  the  item.  They  could  select  from  among  items  that  were  in  the 
video  (targets)  or  not  in  the  video  (foils).  Following  identification,  they  described  what  was 
happening  in  the  scene  and  answered  questions  about  the  relevant  cues,  threats,  and  challenges. 
This  task  required  Soldiers  to  perceive  information  from  the  scene  and  comprehend  the  meaning 
of  that  infonnation. 

Dependent  variables  in  this  task  were  item  identification  accuracy,  item  placement 
accuracy,  and  completeness  of  description.  Independent  variables  included  whether  the  item  was 
a  target  or  foil. 

Decision-Making  Exercises  (DMX).  Soldiers  read  short  scenarios  about  challenging 
patrol  situations.  Each  scenario  included  a  photo  that  depicted  the  last  moment  in  the  scenario. 
After  reading  a  scenario,  Soldiers  had  two  minutes  to  describe  potential  threats  in  the  scene  and 
explain  the  meaning  of  the  threats.  They  then  answered  questions  about  the  threat-relevant  cues 
and  challenges  presented  in  the  scenarios.  This  task  tested  Soldiers’  causal  reasoning  and  critical 
thinking  skills. 

Dependent  variables  for  this  task  included  elaboration  of  answers  to  include  the  meaning 
and  implication  of  threats,  why  cues  were  relevant,  strategies  used  to  detect  and  mitigate  threats, 
and  challenges  when  attempting  to  detect  threats.  We  evaluated  answers  according  to  the 
Dreyfus  and  Dreyfus  5-stage  model  of  skill  acquisition  by  examining  Soldier  explanation  of 
cues,  strategies,  and  challenges  along  criteria  matching  novice,  advanced  beginner,  competent, 
proficient,  and  expert  (Dreyfus  &  Dreyfus,  1986). 

Analysis. 

The  primary  purpose  of  the  research  was  to  evaluate  the  effectiveness  of  the  TDST  by 
measuring  several  metrics  of  perfonnance.  We  used  qualitative  and  quantitative  analysis 
techniques  to  provide  empirical  results  with  both  theoretical  and  applied  value.  We  used  the 
statistical  program  SPSS  to  analyze  the  quantitative  data.  We  conducted  repeated  measures 
analyses  of  variance  (ANOVA)  to  analyze  data  gathered  from  all  the  tasks.  For  all  ANOVAs, 
equality  of  variance  was  assessed  using  Box’s  M  statistic.  There  were  no  observed  violations 
variance-covariance  equality,  all  ps  >  .026. 1  No  statistically  significant  main  effects  of  training 
were  observed  across  tests  (only  one  p  <  .04,  all  other  ps  >  .065);  therefore,  with  one  exception, 
we  do  not  report  main  effects  of  training. 

In  addition,  we  conducted  qualitative  analysis  of  the  data  gathered  in  the  situation 
awareness  task  and  the  DMX  task.  The  purpose  of  the  qualitative  analysis  was  to  identify 


1  Because  Box’s  M  is  a  sensitive  test,  ^-values  >  .001  do  not  represent  a  significant  difference  (Tabachnick  &  Fidell, 
2001). 
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characteristics  in  Soldier  responses  that  resembled  decision  makers  experienced  in  military  threat 
detection  (Zimmerman,  et  ah,  2012).  Two  researchers  rated  select  segments  of  the  situation 
awareness  and  DMX  data  for  each  qualitative  question.  Average  agreement  was  79%  (range 
43%  to  100%).  After  coding  a  sample  of  data,  the  researchers  identified  disagreements  and 
resolved  them  through  discussion.  If  agreement  was  low,  the  researchers  refined  the  rating 
criteria  to  improve  consistency  across  coders  and  then  coded  an  additional  sample  of  data  to 
assess  agreement.  After  achieving  consistent  levels  of  agreement,  each  researcher  coded  half  the 
remaining  data.  The  researchers  occasionally  conferred  with  each  other  to  discuss  difficult 
instances  and  maintain  consistent  coding  practices. 

Results  and  Discussion 


Limited  Threat  Search 

Targets  Identification. 

We  calculated  the  number  of  targets  identified  by  adding  the  frequency  of  clicks  made  by 
each  Soldier  on  targets  that  experienced  Soldiers  identified  in  previous  research  as  threat  relevant 
(Zimmerman,  et  ah,  in  preparation).  A  repeated  measures  ANOVA  using  pre/post  as  a  within- 
subjects  variable  and  training  as  a  between-subjects  variable  revealed  no  interaction  effect  of 
training  x  pre/post  on  the  number  of  threat  relevant  targets  located,  F  (1,  41)  =  .372,  p  =  .545, 
partial  eta2  =  .009.  However,  a  main  effect  of  pre/post  on  number  of  threat  relevant  targets 
located  was  observed,  F  (1,  41)  =  16.143,/?  <  .001 , partial  eta2  =  .283. 2  The  number  of  threat 
relevant  targets  located  increased  from  pre-test  (M  =  36.42,  SD  =  9.94)  to  post-test  (M=  42. 19, 
SD  =  9.85).  Soldiers’  ability  to  locate  threats  in  the  images  improved  over  time  regardless  of  the 
training  intervention,  which  indicates  a  practice  effect  over  time. 

Number  of  Responses. 

No  interaction  effect  of  training  x  pre/post  on  the  total  number  of  responses  (clicks)  was 
observed,  F(l,  41)  =  2.251,  p  =  .141,  partial  eta "  =  .052.  However,  a  main  effect  of  pre/post  on 
the  total  number  of  searches  (clicks)  was  observed,  F  (1,  41)  =  23.766,  p  <  .001 ,  partial  eta 2  = 
.367.  The  number  of  total  searches  (clicks)  increased  from  pre-test  (M  =  141.19,  SD  =  19.42)  to 
post-test  (M=  152.07,  SD  =  15.88).  This  finding  suggests  that  participants  were  likely  to 
respond  more  often  by  clicking  on  threat  locations  over  time,  irrespective  of  the  training 
intervention. 

Proportion  of  Targets  Located  Per  Response  (Hits  Per  Click). 

No  interaction  effect  of  training  x  pre/post  on  the  proportion  of  targets  found  per  response 
(click)  was  observed,  F  ( 1 ,  41)  =  1.897,/?  =  .176,  partial  eta2  =  .044.  A  possible  main  effect  of 
pre/post  on  the  proportion  of  targets  found  per  response  was  observed,  F  (1,  41)  =  4.789,/?  = 


2  Because  we  conducted  four  distinct  analyses  within  this  test,  we  compared  all  statistics  to  a  Bonferroni  corrected 
alpha  of  .0125. 
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.034, partial  eta  =  .  105.  However,  this  effect  failed  to  meet  Bonferroni’s  corrected  alpha  of 
.0125  (correcting  for  four  planned  comparisons).  Despite  failing  to  meet  a  more  stringent  test  of 
statistical  significance,  the  number  of  targets  found  per  response  (clicks)  increased  from  pre-test 
(M=  .26,  SD  =  .06)  to  post-test  (M=  .28,  SD  =  .08).  Soldiers  tended  to  identify  threats  more 
often  during  the  post-test  and  they  tended  to  find  more  targets  per  click.  This  might  indicate  that 
they  became  more  efficient  at  detecting  relevant  threats  over  time,  but  this  claim  should  be 
balanced  against  the  size  of  the  difference  observed  (M=  .02)  and  the  probability  of  error  (p  = 
.034). 


Response  time  (in  milliseconds). 

To  obtain  response  time,  we  calculated  the  time  that  elapsed  between  the  start  of  each 
trial  and  each  response  within  that  trial.  A  repeated  measures  ANOVA  using  pre/post  (pre-  vs. 
post-test)  as  a  within-subjects  variable  and  training  (training  vs.  no  training)  as  a  between- 
subjects  variable  revealed  no  interaction  effect  of  training  x  pre/post  on  response  time,  F  (1,  41) 
=  1 .573  ,p  =.217,  partial  eta2  =  .037.  However,  a  main  effect  of  pre/post  on  response  time  was 
observed,  F  (1,  41)  =  18.283,/?  <  .001 ,  partial  eta “  =  .308.  The  time  participants  took  between 
search  clicks  decreased  from  pre-test  (M=  707.43,  SD  =  214.12)  to  post-test  (M=  602.55,  SD  = 
195.70).  This  result  supports  an  increase  in  efficiency  for  detecting  threats  over  time:  Soldiers 
searched  more  quickly,  more  often,  and  found  more  targets  during  post-test  trials  than  during 
pre-test  trials.  A  possible  main  effect  of  training  was  observed,  t( 41)  =  2.106,/?  =  .041,  Cohen ’s 
d  =  0.64;  Mean  Trained  =  543.5,  SD  =  168.8;  Mean  Untrained  =  664.4,  SD  =  206.6.  There  was 
no  difference  between  groups  pre-training:  t( 41)  =  .913,/?  =  .366.  This  suggests  that  Soldiers  in 
the  training  condition  responded  quicker  than  did  Soldiers  in  the  control  condition  during  the 
limited  threat  search.  However,  this  effect  failed  to  meet  Bonferroni’s  corrected  alpha  of  .016 
(correcting  for  three  planned  comparisons  of  training  group  by  pre/post  -test),  and  so  should  be 
interpreted  with  caution.  That  is,  any  increased  in  efficiency  observed  was  likely  driven  by 
practice  effects  rather  than  training. 

Findings  from  the  limited  threat  search  analysis  indicate  that  training  did  not  influence 
perfonnance  beyond  the  benefits  seemingly  gained  through  practice  alone.  Important  to  note  is 
that  training  did  not  appear  to  hann  performance.  The  tasks  in  the  pre-  and  post-test  conditions 
were  similar  to  the  tasks  in  the  training  condition,  thus  these  results  may  indicate  that  simply 
practicing  threat  detection  using  tasks  such  as  these  are  enough  to  improve  performance. 

Dynamic  Change  Detection 

Accuracy. 

We  considered  Soldier  responses  accurate  when  they  clicked  within  50  pixels  of  the 
change  location  in  the  photo.  Accuracy  was  the  proportion  of  changes  identified  across  20 


3 

In  Zimmerman  et  at,  (2012),  Soldiers  were  83%  accurate  when  clicking  on  target  locations  when  the  accepted  hit 
radius  was  50  pixels.  This  rate  improved  marginally  to  84%  when  the  accepted  hit  radius  increased  to  either  100  or 
150  pixels.  A  50-pixel  radius  provided  a  slightly  more  stringent  test  of  accuracy  without  sacrificing  validity. 
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trials.  A  repeated  measures  ANOVA  using  pre/post  (pre-  vs.  post-test)  as  a  within-subjects 
variable  and  training  (training  vs.  no  training)  as  a  between-subjects  variable  revealed  no 
interaction  effect  of  training  x  pre/post  on  accuracy,  F  (1,  38)  =  1 .49.  p  =  .229,  partial  eta ~  = 
.038.  However,  a  main  effect  of  pre/post  on  accuracy  was  observed,  F  (1,  38)  =  10.04,  p  =  .003, 
partial  eta2  =  .209;  post  hoc  t  (39)  =  3.092 ,p  =  .004,  d  =  0.44. 4  The  number  of  changes 
identified  increased  from  pre-test  (M=  .472,  SD  =  .175)  to  post-test  (M=  .546,  SD  =  .159).  This 
finding  indicates  that  Soldier  ability  to  detect  differences  between  pairs  of  photos  improved  with 
time,  regardless  of  training  intervention.  However,  this  measure  of  accuracy  may  have  been 
somewhat  approximate  because  it  was  measured  using  a  50-pixel  threshold  and  did  not  account 
for  trials  in  which  a  Soldier  may  have  identified  the  change  location  but  simply  did  not  click 
close  enough  to  the  location  on  screen,  in  which  case  it  was  scored  as  inaccurate.  To  assess  this, 
we  analyzed  the  distance  between  response  location  and  change  location. 

Distance. 

To  conduct  a  more  sensitive  analysis  of  response  accuracy,  we  calculated  the  distance 
between  response  location  and  the  center  of  the  change  location.  A  repeated  measures  ANOVA 
using  pre/post  (pre-  vs.  post-test)  as  a  within-subjects  variable  and  training  (training  vs.  no 
training)  as  a  between-subjects  variable  revealed  no  interaction  effect  of  training  x  pre/post  on 
the  distance  between  response  location  and  change  location,  F  (1,  38)  =  0.243,  p  =  .625,  partial 
eta ‘  =  .006.  However,  a  main  effect  of  pre/post  on  distance  was  observed,  F  (1,  38)  =  6.49,  p  = 
.015,  partial  eta2  =  .146;  post  hoc  t  (39)  =  2.55,  p  =  .015,  d  =  0.415.  The  distance  between  the 
response  location  and  the  change  location  decreased  from  pre-test  ( M=  155,  SD  =  51)  to  post¬ 
test  (M=  134,  SD  =  5 1).  This  test  supported  the  accuracy  results  reported  above:  Soldier  ability 
to  identify  the  location  of  changes  improved  over  time,  regardless  of  training  intervention. 

We  conducted  additional  analysis  to  understand  if  training  influenced  Soldier  ability  to 
spot  threat-relevant  changes  more  than  the  ability  to  spot  threat-irrelevant  changes. 

Threat  relevant  changes.  Analysis  of  only  threat  relevant  changes  revealed  no 
interaction  effect  of  training  x  pre/post  on  accuracy,  F  (1,  38)  =  .029,  p  =  .867 ,  partial  eta “  = 
.001. 


Threat  irrelevant  changes.  Analysis  of  only  threat  irrelevant  changes  revealed  no 

2 

interaction  effect  of  training  x  pre/post  on  accuracy,  F  ( 1 ,  38)  =  1.994,/?  =  .166,  partial  eta "  = 


.050. 


Threat  relevant/irrelevant  comparison.  We  analyzed  the  interaction  of  training,  pre/post 
testing,  and  the  threat  relevance  of  the  change  in  each  pair  of  photos  by  calculating  difference 
scores  at  each  test  time  (pre  vs.  post)  and  using  the  scores  as  the  dependent  variable  in  a  repeated 
measures  ANOVA.  Analysis  of  response  time  difference  (RTirreieVant  -  RTReievant)  revealed  no 
interaction  between  training  x  pre/post,  F  (1,  38)  =  .484,  p  =  .491,  partial  etaz  =  .013.  Analysis 


4  Because  we  conducted  two  distinct  analyses  (accuracy  and  distance)  within  this  test,  we  compared  all  statistics  to  a 
Bonferroni  corrected  alpha  of  .025. 
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of  accuracy  difference  (ACCReievant  -  ACCirreievant)  also  revealed  no  interaction  between  training  x 
pre/post,  F  ( 1 ,  38)  =  .  149,  p  =  .702,  partial  eta2  =  .004.  These  findings  indicate  that  threat 
relevance  did  not  influence  reaction  time  or  accuracy.  Overall,  the  only  statistically  significant 
effects  observed  in  the  change  detection  task  might  indicate  practice  effects,  as  Soldier  ability  to 
detect  changes  across  pairs  of  photos  improved  over  time,  but  not  with  training. 

These  results  mirror  those  found  in  the  limited  threat  search  task  by  again  indicating  that 
practicing  the  skills  required  to  accomplish  the  task  was  enough  to  improve  performance,  with 
no  benefit  exclusively  from  training.  Again,  training  did  not  appear  to  harm  perfonnance;  it 
simply  did  not  have  an  observable  effect  on  perfonnance. 

Situation  Awareness  -  Level  1:  Perception 

Target  selection. 

To  assess  threat  detection  perception,  we  analyzed  the  number  of  target  and  foil  items 
selected  in  the  wireframe  images.  Repeated  measures  ANOVAs  with  pre/post  (pre-  vs.  post-test) 
as  a  within- subject  factor  and  training  (training  vs.  no  training)  as  a  between-subjects  factor 
revealed  no  interaction  effect  of  training  x  pre/post  on  the  number  of  targets  (vs.  foils)  selected, 

F  (1 ,  3 1)  =  .00 1 ,  p  =  .977 ,  partial  eta 2  <.001.  No  main  effect  of  pre/post  on  the  number  of 
targets  selected  was  observed,  F  ( 1 ,  31)  =  .911,  p  =  .346,  partial  eta  =  .029.  No  interaction 
effect  of  training  x  pre/post  on  the  number  of  foils  (vs.  targets)  selected  was  observed,  F  (1,  3 1) 

=  .111,  p=  .387,  partial  eta  =  .024.  However,  a  main  effect  of  pre/post  on  the  number  of  foils 
selected  was  observed,  F  ( 1 ,  31)  =  8.228 ,p  =  .007 ,  partial  eta 2  =  .210.  The  number  of  foils 
selected  decreased  from  pre-test  (M=  3.48,  SD  =  3.4)  to  post-test  (M=  2.39,  SD  =  3.0). 
Collapsing  across  training  conditions,  two  paired  samples  t-tests  revealed  a  difference  in  the 
number  of  targets  selected  versus  the  number  foils  selected:  a  pre-test,  more  targets  (M=  6.79, 
SD  =  2.27)  than  foils  (M=  3.48,  SD  =  3.41)  were  selected,  t( 32)  =  6.94 ,p  <  .001,  d=  1.31; 
similarly,  at  post-test,  more  targets  (M=  6.42,  SD  =  2.45)  than  foils  (M=  2.39,  SD  =  2.97)  were 
selected,  t( 32)  =  8.40,/?  <  .001,  d  =  1.48.  These  findings  indicate  that  Soldiers  were  no  more 
likely  to  select  targets  over  time  but  they  did  appear  to  discriminate  by  selecting  fewer  foils  over 
time.  This  discrimination  does  not  appear  to  be  a  function  of  training. 

Target  vs.  foil  selection. 

Given  the  difference  between  selecting  targets  versus  foils  over  time,  we  calculated  a 
difference  score  for  this  factor  and  included  it  as  a  dependent  variable  in  a  repeated  measures 
ANOVA.  Analyzing  the  difference  between  the  number  of  targets  selected  versus  the  number  of 
foils  selected  revealed  no  interaction  effect  of  training  x  pre/post,  F  (1,  31)  =  .974,/?  =  .331, 
partial  eta 2  =  .030.  However,  a  main  effect  of  pre/post  on  target  vs.  foil  selection  was  observed, 
A  ( 1 ,  3 1)  =  4.28 1 , /?  =  .047, partial  eta "  =  .121.  However,  this  effect  failed  to  meet  Bonferroni’s 
corrected  alpha  of  .025.  Despite  failing  to  meet  a  more  stringent  test  of  statistical  significance, 


5  Because  we  conducted  two  distinct  analyses  within  this  test,  we  compared  all  statistics  to  a  Bonferroni  corrected 
alpha  of  .025. 
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the  difference  in  the  number  of  targets  (vs.  foils)  selected  appears  to  have  increased  from  pre-test 
(M=  13.30,  SD  =  2.73)  to  post-test  (M=  14.03,  SD  =  2.76).  This  finding  further  indicates  that 
while  the  number  of  targets  selected  remained  relatively  stable  over  time,  the  number  of  foils 
selected  decreased  over  time  (a  5%  increase  in  targets  selected  vs.  a  31%  decrease  in  foils 
selected). 

As  observed  in  the  previous  tasks,  practice  effects  seemed  to  account  for  observed 
difference  in  performance.  This  is  not  to  say  that  training  was  ineffective.  It  did  not  hann 
perfonnance;  however,  its  effects  are  at  least  matched  by  those  of  practice  alone. 

Situation  Awareness  -  Level  2:  Comprehension. 

To  understand  Soldier  comprehension,  we  conducted  both  quantitative  and  qualitative 
analysis  of  Solder  responses  to  four  questions  (Appendix  A).  We  expected  that  Soldiers  who 
completed  training  would  provide  more  elaborate  descriptions  and  would  provide  details  that 
indicated  they  comprehended  the  meaning  and  implications  of  the  threat  cues.  Across  all 
questions,  Soldiers  gave  relatively  brief  responses. 

Situation  comprehension. 

Researchers  rated  answers  to  the  first  question  “Describe  what  is  happening  in  the  scene” 
as  literal  or  implied.  Responses  received  literal  ratings  when  Soldiers  described  what  they 
viewed  in  the  videos  with  no  interpretation  of  what  they  saw.  Researchers  rated  responses  as 
implied  when  Soldiers  provided  interpretations  of  what  they  viewed  in  the  videos.  In  previous 
research,  more  experienced  Soldiers  constructed  rich,  causal  stories  about  a  threat  situation 
(Zimmerman,  et  ah,  2012).  We  expected  training  to  enhance  Soldiers’  ability  to  create  causal 
stories  about  the  threat  environment.  For  example,  one  video  clip  showed  Soldiers  walking 
down  a  road  lined  with  high  grass  and  then  walking  down  a  road  in  a  line  with  foreign  personnel 
(for  example,  the  Afghan  National  Army).  Typical  literal  responses  to  this  question  included: 

•  “Foot  patrol”  or  “patrols,” 

•  “A  foot  patrol  is  walking  next  to  a  wall,”  and 

•  “Soldiers  moving.” 

In  such  examples,  Soldiers  described  what  occurred  in  the  video  clip  without  providing 
any  interpretation.  This  is  consistent  with  more  novice  perfonnance,  where  such  responses  are 
generally  fact-driven.  In  general,  Soldiers  with  more  experience  tend  to  create  stories  around 
situations.  Soldiers’  who  responded  with  elaborative  descriptions  often  included  implied  or 
story-telling  components: 

•  “EOD  under  escort  on  a  move  to  a  possible  IED  location,” 

•  “Training  the  Afghan  army,”  and 

•  “Marines  conduct  a  presence  patrol.” 

These  examples  illustrate  Soldier  ability  to  reason  from  what  they  saw  to  interpretations 
based  on  cues  and  the  expectations  that  have  about  the  cues.  We  expected  that  with  training, 


17 


Soldiers  would  provide  answers  that  indicate  better  comprehension  of  the  scenario.  Repeated 
measures  ANOVA  with  pre/post  (pre-  vs.  post-test)  as  a  within-subjects  factor  and  training 
(training  vs.  no  training)  as  a  between-subjects  factor  revealed  no  interaction  effect  on  the 
number  of  literal  vs.  implied  responses  F(l,  32)  =  .13,/?  =  .12, partial  eta2  <  .01.  Soldiers 
descriptions  did  not  vary  with  training,  F  (1,  32)  =  .49,/?  =  .49,  partial  eta “  =  .02  or  pre/post,  F 
(1,  32)  =  2.09,/?  =  .16,  partial  eta' 2  =  .06. 

Threat  identification. 

The  majority  of  Soldiers  responded  to  the  second  question,  “Which  element(s)  in  the 
scene  are  the  biggest  threat?”  by  listing  the  threats  as  asked.  This  is  consistent  with  novice  or 
advanced  beginner  expected  perfonnance  (Dreyfus  &  Dreyfus,  1986;  Zimmerman  et  ah,  2012). 

A  few  Soldiers  also  provided  cursory  explanations  about  the  implications  of  these  threats  (e.g., 
could  be  an  improvised  explosive  device  [IED]).  Across  all  videos,  Soldiers  listed  threats  and 
added  additional  explanation  in  33  instances.  Training  did  not  appear  to  influence  Soldier’s 
propensity  to  elaborate  beyond  threat  identification.  One  scenario  produced  the  most  elaboration 
(8  instances,  see  Figure  5).  In  this  video,  a  group  of  Soldiers  walked  past  a  building,  looked  at 
the  ground,  and  indicated  footprints.  The  terrain  was  rocky,  with  small  hills  that  included  grass 
and  dry  brush.  Soldiers’  hypotheses  included: 

•  “The  box  right  next  to  the  tree  [is  a  threat]  because  it  could  be  an  IED,  the  actual  tree 
[could  be  a  threat]  because  somebody  could  be  hiding  in  it,” 

•  “The  weeds  and  brush  can  hide  IED’s,”  and 

•  “The  brush  and  ditches  can  be  used  to  conceal  an  IED.” 


Figure  5.  Snapshot  of  the  video  that  produced  elaborative  responses. 
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More  elaborative  responses  to  this  video  may  have  been  due  to  unique  features  of  the 
situation.  The  cues  present  in  the  scenario  could  have  been  particularly  strong  (e.g.,  Soldiers  in 
the  video  are  looking  at  the  ground;  they  must  be  looking  for  threats)  making  it  easier  for 
Soldiers  to  give  additional  information.  The  majority  of  Soldiers  described  threat  cues  without 
further  explanation.  For  scenario  described  above,  these  responses  included: 

•  “The  unsettled  dirt,” 

•  “Grassy  area,”  and 

•  “Weapons  not  held  properly.” 

We  hypothesized  that  training  would  influence  Soldier  ability  to  detect  relevant  cues. 
Soldier  identification  of  the  biggest  threat  in  each  video  yielded  two  measures  of  threat  cue 
selection:  the  type  of  cues  and  the  number  of  cues  identified.  On  average,  Soldiers  identified  1.5 
cues  per  video  clip  and  these  cues  ranged  from  seven  to  21  cue  categories  per  video  clip. 
Zimmerman  et  ah,  (2012)  found  that  Soldiers  with  more  deployments  identified  a  greater  variety 
of  threats  compared  to  Soldiers  who  had  never  deployed.  Given  the  relatively  low  level  of 
experience  among  Soldiers  in  this  reseearch,  we  did  not  expect  to  see  significant  differences  in 
cue  identification  with  and  without  training.  However,  we  expected  small  differences  in  the 
kinds  of  cues  selected  by  trained  Soldiers  compared  with  untrained  Soldiers.  In  general,  Soldiers 
did  not  identify  different  cues  as  a  function  of  training  (Table  5). 

Table  5 

Video  Clip  Example  1:  The  Number  of  Soldiers  Who  Discussed  Each  Type  of  Threat  by 
Training  Condition 


Pre-test 

Post-test 

Post-test 

No  Training 

Pre-test  Training 

No  Training 

Training 

Threat  Cue 

(h*  =  11) 

(n  =  6) 

(»  =  7) 

(n  =  13) 

Building 

8 

1 

4 

5 

Grass 

6 

4 

3 

6 

Corn 

0 

1 

2 

1 

Hole  in  wall 

1 

1 

1 

0 

Wall 

1 

0 

2 

1 

Ditch 

0 

0 

0 

1 

IEDs 

0 

0 

1 

2 

Enemy  fire 

1 

0 

0 

0 

*Note:  n  indicates  the  number  Soldiers  per  condition.  Soldiers  could  discuss  more  than  one  type  of 


threat. 


Regardless  of  training,  Soldiers  primarily  identified  the  building  and  the  grass  as  the 
greatest  threats.  Differences  among  the  relative  percentages  of  threats  could  be  due  to  variation 
in  the  number  of  Soldiers  who  responded  in  each  condition.  In  another  video  example,  Soldiers 
identified  the  threats  in  a  video  that  depicted  U.S.  military  and  foreign  military  personnel 
walking  down  a  road.  A  motorcycle  drove  past  the  forces  while  they  walked  down  the  middle  of 
the  road.  Again,  Soldiers  identified  similar  cues  regardless  of  training  (Table  6). 
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Table  6 


Video  Clip  Example  2:  The  Number  of  Soldiers  Who  Discussed  Each  Type  of  Threat  by 
Training  Condition 


Pre-test 

Pre-test 

Post-test 

Post-test 

No  Training 

Training 

No  Training 

Training 

Threat  Cue 

(n*  =  7) 

( n  =  14) 

(n  =  10) 

(»  =  5) 

IED 

0 

0 

1 

0 

Vehicle 

1 

7 

5 

2 

Man  on  bike 

4 

5 

3 

3 

White  van 

1 

0 

1 

0 

Hole  in  ground 

1 

0 

1 

0 

People 

0 

0 

2 

1 

Moped 

3 

4 

0 

0 

VBIED 

0 

2 

1 

0 

Fresh  dirt 

0 

1 

0 

0 

Light  pole 

0 

1 

0 

0 

Stack  of  Boxes 

0 

1 

0 

0 

*Note:  n  indicates  the  number  Soldiers  per  condition.  Soldiers  could  discuss  more  than  one  type  of 


threat. 


Across  training  conditions,  Soldiers  identified  vehicles  and  the  man  on  the  bike  as  the 
two  greatest  threats.  After  training,  Soldiers  appear  to  identify  fewer  threats;  however,  this  could 
be  due  to  a  small  number  of  Soldiers  responding  in  each  condition.  An  alternative  explanation  is 
that  with  training,  participants  narrow  their  choices  to  threat  relevant  cues  rather  than  identifying 
all  possible  threats  (Zimmerman  et  ah,  2012).  Such  a  finding  may  be  scenario  specific,  as  this 
finding  did  not  appear  to  generalize  across  scenarios. 

If  training  helped  Soldiers  focus  on  more  relevant  cues,  we  would  expect  a  difference  in 
the  number  of  relevant  cues  reported.  Repeated  measures  ANOVA  with  pre/post  (pre-  vs.  post¬ 
test)  as  a  within-subject  factor  and  training  (training  vs.  no  training)  as  a  between-subjects  factor 
revealed  no  interaction  effect  on  the  number  of  reported  cues,  F  (1,  33)  =  .25,  p  =  .62,  partial 
eta 2  =  .01.  There  was  no  training  effect  on  the  number  of  reported  cues,  F  (1,  33)  =  .21  ,p  =  .65, 
partial  eta  =  .01,  and  the  number  of  cues  reported  did  not  differ  from  pre-  to  post-training,  F  (1, 
33)  =  1.03 ,p  =  .32, partial  eta '  =  .03. 

Threat-related  story  building. 

When  responding  to  the  question  “What  makes  you  believe  that  these  elements  are  a 
threat,”  Soldiers  tended  to  provide  general  explanations,  often  restating  the  cues  provided  in 
response  to  the  second  question  (“Which  element(s)  in  the  scene  is  the  biggest  threat”),  or 
matching  the  cue  to  an  associated  threat  with  no  additional  explanation.  Researchers  classified 
responses  into  three  categories:  1)  general  cue  or  no  explanation,  2)  simple  cue/threat  match 
with  no  elaboration,  3)  simple  cue/threat  match  with  elaboration.  For  example,  Soldiers  watched 
a  video  that  depicted  military  personnel  walking  down  a  street  (see  Figure  6).  The  street 
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contained  numerous  vehicles,  trees,  power  lines,  and  buildings.  Three  participants  restated  the 
cues  named  in  in  response  to  the  previous  question: 

•  “Everything  on  the  street,” 

•  “They  are  hostile,”  and 

•  “They  are  just  sitting  around.” 

These  Soldiers  did  not  explain  why  they  thought  specific  cues  were  threats,  instead  they 
provided  details  about  the  cues  or  provided  general  non-specific  answers  cues  (e.g.,  everything  is 
a  threat,  or  people  are  hostile).  This  type  of  explanation  is  typical  of  novices  who  often  struggle 
to  explain  the  relevance  of  the  cues.  The  majority  of  responses  to  this  video  were  simple 
cue/threat  matches  with  no  elaboration.  This  indicated  that  Soldiers  were  able  to  identify  the 
threats  associated  to  relevant  cues,  but  not  extend  this  thinking  to  make  predictions  or  strategize. 
Simple  cue/threat  matches  are  characteristic  of  advanced  beginners,  who  are  able  to  make 
relevant  distinctions  between  a  cue  and  an  outcome  as  they  build  their  scripts  of  situation  (e.g., 
Dreyfus  &  Dreyfus,  1986).  Typical  answers  included: 


•  “Any  of  the  [cars]  could  be  VBIEDs,” 

•  “IED  can  be  hidden  in  the  engine  compartment  of  the  blue  truck  and  the  small  wall 
can  provide  cover  for  the  IED,”  and 

•  “Snipers  can  be  in  the  tower  and  bombs  could  be  in  vehicles,  on  people,  or  on  the  side 
of  the  road.” 


Figure  6.  Snapshot  of  the  video  that  corresponds  to  explaining  why  the  elements  present  a 
threat. 
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Some  Soldiers  demonstrated  the  ability  to  think  beyond  simple  cue/threat  matches  and 
elaborated  on  their  answers  by  including  enemy  strategy,  challenges,  and  story-building. 

Soldiers  demonstrated  thinking  characteristic  of  competent  skill  levels  (Dreyfus  &  Dreyfus, 
1986).  Examples  included: 

•  “Open  road  implies  a  location  for  VBIEDs  and  IEDs,  concealment  hides  enemy 
combatants  setting  up  an  ambush,” 

•  “There  are  so  many  cars  that  [this  situation]  could  easily  be  a  set  up  for  a  complex 
attack,”  and 

•  “They  stopped  there  for  a  reason,  maybe  that’s  what  the  enemy  wants.” 

These  responses  indicated  that  Soldiers  were  hypothesizing  about  the  enemy  activities,  in 
addition  to  matching  the  cue  to  a  threat. 

Despite  qualitative  evidence  that  Soldiers  responded  in  ways  that  represent  differing 
levels  of  skill  development,  training  had  no  effect  on  Soldier  explanations  of  why  the  elements 
were  threats,  F  (1,  32)  =  .84,  p  =  .36,  partial  eta  =  .03.  There  was  no  training  by  pre/post 
interaction,  F  (1,  32)  =  .14 ,p=  .11, partial  etaz  <  .01,  nor  was  there  a  main  pre/post  effect,  F  (1, 
32)  =  3.10 ,p  =  .09, partial  eta 2  =  .09. 

Threat  detection  challenges. 

Soldiers  answered  the  question,  “What  are  the  challenges  to  threat  detection  in  this  scene 
(or  what  makes  it  difficult)?”  with  varying  levels  of  detail.  Previous  research  showed  that 
Soldiers  with  less  experience  had  more  difficulty  discussing  challenges  to  threat  detection 
(Zimmerman  et  ah,  2012).  Given  the  experience  level  represented  in  this  sample,  we  expected 
Soldiers  to  have  difficulty  discussing  the  challenges  to  threat  detection.  Researchers  categorized 
responses  into  three  categories:  (a)  decontextualized  challenge,  (b)  situational  challenge,  but 
general  or  procedural,  (c)  creating  a  story  around  the  challenge.  Overall,  Soldiers’  provided 
general  or  procedural  challenges,  either  in  the  context  of  the  video  situation  or  in  general.  The 
following  response  examples  correspond  to  a  video  clip  showing  a  Soldier  asking  a  local  civilian 
to  open  the  hood  of  his  car,  followed  by  the  Soldier  looking  in  the  car’s  engine  compartment 
(Figure  7). 
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Figure  7.  Snapshot  of  the  video  that  corresponds  to  responses  about  challenges  to  threat 
detection. 

Responses  categorized  as  decontextualized  challenge  reflected  procedural  or  general 
challenges  that  could  apply  to  all  scenarios.  These  responses  reflect  novice  perfonnance  that  is 
“context-free”  and  objectively  factual,  consistent  with  Dreyfus  and  Dreyfus’s  (1986)  model  of 
skill  acquisition.  These  responses  included: 

•  “Possible  suspect,” 

•  “The  terrain,  and  not  knowing  what  to  look  for,”  and 

•  “Undetectable.” 

Soldiers  often  provided  descriptions  of  challenges  specific  to  the  situation  but  these 
challenges  tended  to  be  procedural  in  nature.  These  responses  were  specific  to  the  situation  and 
indicated  the  Soldiers  were  able  to  identify  relevant  challenges  (e.g.,  safely  inspecting  the  car, 
dividing  attention  between  important  cues,  and  identifying  IEDs  in  the  vehicle).  As  experience 
increases,  Soldiers  should  begin  to  differentiate  between  general,  context-free  threats,  and  those 
that  are  specific  to  the  immediate  situation.  Responses  in  this  category  included: 

•  “The  Soldier  cannot  safely  check  the  car  himself,” 

•  “Keeping  an  eye  both  on  the  civilian  and  car,”  and 

•  “The  vast  amount  of  places  in  the  vehicle  that  an  IED  could  be  placed.” 

The  final  category  included  responses  that  included  stories  about  challenges  that  went 
beyond  the  typical  situational  script.  In  one  example,  the  Soldier  said,  “can’t  really  see  the 
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windows  in  the  building  next  to  [the  Soldier],  the  car  can  have  hidden  wires,  the  [local  civilian] 
has  not  been  searched,  which  could  be  hard  to  get  the  man  to  do.”  In  this  example,  the  Soldier 
identified  two  challenges  that  fit  into  the  previous  category:  1)  being  unable  to  see  in  the 
windows  and  the  car  having  hidden  wires  and  2)  the  possibility  that  the  man  was  not  searched. 
Projecting  beyond  these  challenges,  the  Soldier  also  considered  future  difficulties  associated  with 
gaining  consent  to  search  the  man.  This  Soldier  reasoned  beyond  the  man  not  being  searched  (a 
procedural  concern)  and  discussed  how  interacting  with  the  civilian  and  the  search  could  be 
problematic.  Although  the  Soldier  did  not  describe  why  this  was  a  challenge,  the  comment 
reflects  thinking  beyond  the  immediate  challenge  and  incorporates  contextual  elements. 

An  example  from  another  video  clip  included,  “the  smoke  coming  from  the  bushes  takes 
[Soldiers’]  attention  away  from  the  car  and  the  trash  pile.”  Here,  the  Soldier  demonstrated  that 
he  was  considering  enemy  strategy  (e.g.,  the  smoke  diverts  Soldier  attention  from  the  real  threat). 
This  response  was  specific  to  the  immediate  situation  and  accounted  for  relevant  contextual 
information  that  could  increase  the  challenges  to  threat  detection. 

Although  Soldiers’  responses  reflected  different  levels  of  expertise,  repeated  measures 
ANOVA  with  pre/post  (pre-  vs.  post-test)  as  a  within-subject  factor  and  training  (training  vs.  no 
training)  as  a  between-subjects  factor  revealed  no  interaction  effect  on  descriptions  of  challenges 
F  (1,  32)  =  .10 ,p  =  .15, partial  eta 2  <  .01.  Training  did  not  affect  Soldier  ability  to  create  a  story 
about  the  challenge,  F  ( 1 ,  32)  =  2.80.  p  =  AO,  partial  eta  =  .08  and  there  was  no  change  from 
pre-  to  post-training,  F  (1,  32)  =  1.33 ,p  =  .26,  partial  eta2  =  .04. 

Decision-Making  Exercises 

Soldiers  read  two  decision-making  exercises  (DMX)  pre-test  and  two  DMXs  post-test. 
They  answered  four  questions  after  reading  each  scenario  (Appendix  B).  From  these  answers,  we 
conducted  qualitative  and  quantitative  analyses.  Below  we  present  the  results  from  each 
question. 

Threat  descriptions  and  implications. 

After  reading  the  DMX  scenario,  Soldiers  answered  a  question  designed  to  uncover  the 
threats,  cues,  and  interpreted  meanings  of  these  threats.  The  first  question  asked  Soldiers  to 
“Describe  any  potential  threats  present  in  this  scene  and  explain  the  meaning  and  implications  of 
these  threats.  You  have  two  minutes  to  write  a  response.”  Researchers  coded  responses 
according  to  a  data  category  scheme  developed  by  Zimmerman  et  al.  (2012).  In  that  previous 
analysis,  researchers  classified  interview  data  into  seven  categories:  types  of  threats,  threat  cues, 
strategies  for  threat  detection,  threat  detection  tasks,  threat  detection  skills,  challenges  in  threat 
detection,  and  solutions  for  threat  detection  preparation/training.  Zimmerman  et  al.  found  that 
Soldiers  differed  in  their  ability  to  produce  data  for  each  category  based  on  their  experience 
levels.  Soldiers  with  a  greater  number  of  deployments  more  often  discussed  strategies  for  threat 
detection,  threat  detection  skills,  challenges  in  threat  detection,  and  solutions,  compared  to 
Soldiers  who  had  never  deployed.  All  Soldiers,  regardless  of  deployment  experience,  discussed 
types  of  threats,  threat  cues,  and  threat  detection  tasks.  Using  this  coding  scheme,  we 
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determined  if  training  changed  Soldier’s  response  characteristics  to  match  more  closely  the 
responses  of  experienced  Soldiers  in  the  previous  research. 

In  the  pre-test,  no  training  condition,  Soldiers  identified  37  types  of  threats  and  57  threat 
cues,  3  strategies  for  threat  detection,  2  tasks,  and  2  challenges  across  all  four  scenarios.  They 
did  not  identify  any  skills,  or  solutions.  In  the  pre-test,  training  condition,  Soldiers  identified  36 
types  of  threats,  88  threat  cues,  2  strategies,  and  7  challenges  across  all  scenarios.  They  did  not 
identify  any  skills,  tasks,  or  solutions.  Threat  types  included  IEDs,  ambush,  Taliban,  suicide 
bombers,  and  weapons.  Soldiers  often  reported  threat  types  in  combination  with  threat  cues. 
Common  threat  cues  included  people,  parked  cars,  and  men  acting  strange  or  running  towards 
Soldiers.  For  example: 

•  “There  are  many  people  out  on  this  road  moving  rocks  (cue)  which  can  hide  IEDs 
(threat).  Many  of  these  people  (cue)  don’t  look  like  a  threat  but  [it  is]  possible,  could 
be  another  ambush  (threat)  waiting  for  the  right  [moment]  to  set  off  IEDs.  The 
mountains  (cue)  in  front  could  possibly  have  snipers  (threat)  as  well.” 

•  “If  [the  area  is]  very  empty  (cue),  an  IED  (threat)  may  be  placed  and  it  is  in  the 
middle  of  the  city  [which  is]  probably  usually  busy  (cue).  Car  (cue)  parked  there 
could  be  a  vehicle  borne  IED  (threat)  and  the  men  in  the  video  could  be  Taliban 
(threat)  and  have  IEDs  strapped  on  to  them  (threat).” 

The  strategies  Soldiers  reported  included: 

•  “Those  could  be  insurgents  planting  IEDs  into  the  road,  who  are  using  the 
Commander's  Emergency  Response  Program  (CERP)  as  an  excuse  to  be  “working.” 

I  would  not  believe  it  is  legitimate  work  until  I  see  records  of  it.” 

•  “. .  .1  would  stop  and  investigate  and  also  see  if  there  was  any  construction  work  to  be 
done  at  this  time.” 

Challenges  Soldiers  reported  included  not  knowing  a  suspicious  person’s  intentions, 
protecting  innocent  bystanders,  and  not  knowing  if  other  Soldiers  are  aware  of  the  threat. 

In  the  post-test,  no  training  condition,  Soldiers  identified  23  types  of  threats,  48  threat 
cues,  and  5  challenges  across  all  scenarios.  They  did  not  identify  any  strategies,  skills,  tasks,  or 
solutions.  In  the  post-test,  training  condition,  Soldiers  identified  37  types  of  threats,  57  threat 
cues  across  all  scenarios.  They  did  not  identify  any  strategies,  tasks,  skills,  challenges  or 
solutions.  Common  threat  types  included  IEDs,  suicide  bomber,  ambush,  snipers.  Consistent 
with  the  findings  in  the  pre-test  condition,  Soldiers  often  reported  threat  cues  paired  with  threat 
types.  Common  threat  cues  included  cars,  buildings,  running  man,  wheelbarrows,  and  disturbed 
dirt. 


Based  on  previous  research  (Zimmerman  et  ah,  2012),  we  expected  that  training  might 
increase  the  rate  at  which  Soldiers  described  strategies,  challenges,  skills,  and  solutions.  While 
Soldiers  in  the  training  group  identified  the  same  number  of  threats  post-test,  they  provided 
fewer  threat  cues  and  did  not  describe  any  strategies  or  challenges.  Soldiers  in  the  no  training 
group  identified  fewer  threat  types  compared  to  the  training  group  and  compared  to  the  training 
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group  post-test.  Fatigue  may  have  affected  performance  during  the  post-test.  The  lack  of 
changes  in  the  training  group  may  indicate  that  the  training  was  too  subtle  have  an  effect  on 
open-ended  question  responding. 

Explanation  of  Threat  Factors. 

Soldiers  responded  to  the  question,  “List  three  to  five  factors  that  make  this  situation 
dangerous  and  provide  a  brief  explanation  of  why  each  is  dangerous.”  The  majority  of  Soldiers 
(M=  17)  listed  three  factors  per  clip,  thus  we  evaluated  the  first  three  factors  reported  and 
evaluated  differences  across  conditions.  No  Soldiers  listed  five  factors,  and  an  average  of  less 
than  one  (0.75)  Soldier  listed  four  factors  per  clip.  On  average,  five  Soldiers  listed  one  factor  per 
clip  and  six  Soldiers  listed  two  factors  per  clip.  Soldiers  tended  to  identify  threat  cues  in  the 
photographs  that  accompanied  each  DMX  rather  than  threats  described  in  the  written  scenario. 
While  many  of  the  cues  were  the  same,  very  few  Soldiers  related  the  cues  directly  to  the  scenario 
content.  Of  the  134  responses  to  this  question,  only  eight  responses  (6%)  directly  related  the 
cues  to  the  situation  presented  in  the  scenario.  No  differences  emerged  across  pre-  and  post-test 
conditions,  and  we  could  not  draw  any  reliable  conclusions  given  the  small  sample  size. 

Not  all  Soldiers  provided  explanations  about  why  each  factor  was  dangerous.  For 
example,  in  the  Road  Construction  scenario  (Appendix  B),  the  picture  shows  men  working  on  a 
rocky  mountainside  moving  rocks  in  wheelbarrows  across  a  roadway.  The  written  scenario 
depicts  a  situation  where  security  patrols  have  increased  because  of  several  small-arms  attacks 
and  attempted  ambushes.  One  afternoon,  the  unit  discovers  local  civilians  working  on  the  road, 
but  they  have  no  record  of  scheduled  work  in  the  area.  Examples  of  response  identifying  cues 
but  not  providing  explanations  included: 

•  “Concealment,  terrain,  deception,” 

•  “Mound  of  dirt,”  and 

•  “Large  number  of  workers,  large  amount  of  rubble,  unscheduled  construction.” 

Soldiers  who  provided  explanations  about  why  the  cues  they  identified  were  dangerous 
indicated: 

•  “They  might  be  terrorists.  Nice  pile  of  rubble  to  put  IEDs  in.  Good  place  for  an 
ambush.  Might  have  weapons  hidden  in  the  carts.” 

•  “Digging  on  the  side  of  the  road  can  hide  explosives.  Work  being  done  on  a  curve  in 
the  road  [makes  it]  harder  to  see  buried  explosives.  High  elevation  on  both  sides 
[makes  this]  easy  ambush  spot.” 

•  “Wheelbarrows  could  be  hiding  weapons  for  an  ambush.  Holes  on  the  road  side 
[where  they  could  have]  planted  IED’s.  All  military  age  males  [are]  fit  to  fight.  The 
terrain  is  good  for  sniper  attacks.” 

Repeated  measures  ANOVAs  assessing  the  effects  of  training  and  pre/post  tests  on 
whether  Soldiers  provided  explanations  revealed  no  interactions,  F  (1,  29)  =  .00.  p  =  .95,  partial 
eta "  <  .01.  Training  significantly  influenced  whether  Soldiers  gave  an  explanation  about  why  the 
cues  indicated  danger,  F(l,  29)  =  5.41,/?  =  .03, partial  eta  =  .16.  We  expected  that  Soldiers 
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would  provide  more  detailed  explanations  after  training;  however,  untrained  participants  (M  = 
.80,  SD  =  .02)  provided  explanations  more  often  than  did  trained  participants  (M=  .56,  SD  =  .02) 
across  pre/post  tests.  We  do  not  have  an  explanation  for  this  finding.  The  participants  were 
randomly  assigned  to  the  training  conditions  and  the  effect  was  across  the  pre/post  condition, 
indicating  that  training  did  not  necessarily  influence  the  results.  There  were  no  significant 
effects  for  the  pre/post  condition  and  the  training  by  pre/post  interaction  was  not  significant.  The 
observed  power  was  low  (equal  to  .05),  which  could  have  influenced  the  ability  to  detect  a 
significant  effect. 

Threat  Detection  and  Mitigation  Strategies. 

To  understand  how  Soldiers  think  about  threat  response  strategies,  we  asked  them  to 
“Name  the  one  potential  threat  you  are  most  concerned  about.  What  strategies  would  you  use  to 
detect  and  mitigate  that  threat?”  Researchers  identified  the  first  named  threat  and  reviewed  the 
data  to  detennine  if  Soldiers  reported  different  threats  depending  on  training.  There  were  no 
consistent  differences  in  the  threats  discussed  across  the  training  and  the  no  training  groups.  The 
majority  of  the  threats  named  were  the  threat  cues,  such  as  “the  man  running,”  “the  car  parked 
too  close  to  the  checkpoint,”  and  “men  with  wheelbarrows.”  In  35%  of  the  responses  (n  =  33), 
Soldiers  named  the  possible  threat  associated  with  the  threat  cues,  such  as  IEDs,  ambushes,  and 
snipers. 


We  identified  whether  or  not  Soldiers  identified  strategies  to  detect  or  mitigate  the  threat. 
For  example,  the  Afghan  National  Police  (ANP)  Incident  DMX  scenario  (Appendix  B)  includes 
a  picture  showing  an  old  woman,  a  truck  with  two  Afghan  nationals,  and  a  building  located 
behind  the  truck.  The  scenario  describes  a  situation  where  the  Soldier  is  manning  a  post  with  a 
member  of  the  ANP  who  is  acting  erratically  and  threatens  to  call  the  Taliban.  In  this  scenario, 
Soldiers’  responses  where  they  only  identified  the  threat  included: 

•  “Old  woman,” 

•  “The  one  threat  that  I  am  most  worried  about  is  him  waving  his  gun  around,  not 
caring  about  his  muzzle  awareness.  He  could  possibly  shoot  me  or  anyone  around  the 
post  and  start  a  shooting  spree  and  cause  other  members  of  the  ANP  to  start  shooting 
as  well,”  and 

•  “Ambush  from  houses.” 

Examples  of  responses  in  which  Soldiers’  discussed  strategies  include: 

•  “I’d  clear  everyone  out  of  there  and  take  control  of  that  machine  gun,” 

•  “That  old  lady  being  a  possible  suicide  bomber  would  be  my  top  priority.  I  would 
have  a  female  Soldier  come  and  search  her  to  make  sure  she’s  clear.”  and 

•  “Most  concerned  about  the  angry  ANP.  [I  would]  ask  to  have  him  replaced,  removed 
from  the  area.  Keep  a  close  watch  on  him  and  observe  for  any  continued  erratic 
behavior.” 

We  expected  that  trained  Soldiers  should  be  better  able  to  strategize  about  threat 
detection  or  mitigation,  given  that  Soldiers  with  more  experience  are  more  apt  to  provide  threat 
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detection  strategies  when  describing  threat  situations  (Zimmerman  et  ah,  2012).  Repeated 
measures  ANOVA  with  pre/post  (pre-  vs.  post-test)  as  a  within-subject  factor  and  training 
(training  vs.  no  training)  as  a  between-subjects  factor  revealed  no  interaction  effect  on  reporting 
of  strategies,  F(  1,  29)  =  .29,  p  =  .59,  partial  eta2  =  .01.  Observed  power  was  low  (equal  to  .08), 
which  makes  it  difficult  to  detect  an  effect.  Training  had  a  marginally  significant  effect  on 
whether  Soldiers  offered  a  strategy  to  detect  or  mitigate  the  threats,  F  (1,  29)  =  3.59,  p  =  .07, 
partial  eta "  =  .11.  Trained  Soldiers  (M=  .68,  SD  =  .02)  provided  strategies  to  mitigate  the  threat 
more  often  than  did  untrained  Soldiers  (M=  .44,  SD  =  .02).  There  was  a  significant  effect  from 
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pre-test  to  post-test  on  whether  Soldiers  offered  a  strategy,  F{  1,  29)  =  1 1.78,/?  <  .01,  partial  eta 
=  .29,  though  it  was  not  in  the  expected  direction.  Soldiers  offered  strategies  less  often  during 
the  post-test  (M=  .43,  SD  =  .01)  than  they  did  during  the  pre-test  (M=  .69,  SD  =  .01).  This 
finding  could  indicate  fatigue  from  pre-  to  post-test,  though  the  main  effect  of  training  indicates 
that  the  training  group  provided  more  strategies  across  test  sessions.  Greater  power  might  clarify 
if  an  interaction  exists  and  if  it  is  driven  by  the  training  condition  post-test. 

Threat  Detection  Challenges, 

To  identify  Soldiers’  understanding  of  challenges  in  threat  situations,  Soldiers  responded 
to  the  request  to  “List  the  top  three  challenges  to  detecting  threats  in  this  situation  and  state  why 
they  are  challenges.”  Researchers  identified  the  first  three  named  challenges  to  determine 
whether  Soldiers  identified  different  challenges  after  training.  Soldiers  appeared  to  identify 
similar  challenges  both  pre-  and  post-training.  Table  7  depicts  the  challenges  Soldiers’  indicated 
for  the  Road  Construction  DMX  scenario  (Appendix  B).  Across  all  conditions,  Soldiers  reported 
that  detennining  the  work  legitimacy  was  an  important  challenge.  They  also  discussed  the 
mountain/terrain/location  in  which  the  scenario  occurred  and  the  wheelbarrows  or  identification 
of  the  wheelbarrow  contents.  Other  challenges  appear  in  each  condition,  such  as  losing  hearts 
and  minds  of  the  local  population  and  the  absence  of  any  apparent  weapons.  The  training  group 
seemed  to  list  a  greater  variety  of  challenges  post-test  in  this  scenario;  however,  not  all  the 
scenarios  reflected  this  pattern.  For  instance,  in  the  Convoy  Ops  DMX  scenario,  Soldiers  in  the 
no  training  group  reported  a  greater  variety  of  challenges,  and  reported  challenges  more  often 
than  did  the  training  group,  both  pre-  and  post-test. 
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Table  7 


The  Number  of  Challenges  Identified  by  Training  Condition  in  the  Road  Construction 
Scenario 


Pre-test 

Pre-test 

Post-test 

Post-test 

Challenge  Type 

No  Training 
(n  *  =  5) 

Training 
(n  =  10) 

No  Training 
(n  =  8) 

Training 
(n  =  15) 

Work  legitimacy 

1 

3 

1 

2 

Wheelbarrow/identification  of 

1 

1 

1 

contents 

A 

Unknown  enemy 

1 

l 

Mountains/terrain/location 

1 

1 

l 

4 

Men/workers 

l 

3 

Rubble 

1 

Line  of  sight 

l 

Hiding  places 

l 

Losing  hearts  and  minds 

l 

Construction 

l 

Dirt 

l 

Potential  spotters 

l 

Can’t  see  in  holes 

1 

Tools 

1 

No  weapons  apparent 

1 

Number  of  men 

1 

Unidentified  personnel 

1 

*Note:  n  indicates  the  number  of  challenges  identified. 


Further  Analysis 

The  primary  objective  of  this  research  was  to  determine  if  the  TDST  had  an  effect  on 
identified  skills  such  as  target  identification,  detecting  relevant  changes,  and  causal  reasoning. 
To  determine  if  prior  training  influenced  performance  during  this  research,  we  asked  Solders  to 
describe  any  training  they  received  that  improved  their  ability  to  detect  threats  (Appendix  C, 
question  12).  From  those  responses,  16  participants  responded  that  they  had  received  training 
relevant  to  threat  detection,  while  32  responded  that  they  had  not  received  such  training. 

Because  this  was  a  small  and  disproportionate  sample,  inferential  statistical  analysis  would  be 
suspect.  However,  a  descriptive  analysis  did  yield  suggestive  findings  concerning  whether 
previous  training  influenced  performance. 

Soldiers  who  reported  receiving  threat  detection  training  in  the  Army,  prior  to  the 
experiment,  responded  similarly  on  the  dependent  variables  to  Soldiers  who  had  not  received 
previous  training.  Army  training  on  threat  detection  can  take  several  days  to  weeks  to  complete. 
Moreover,  some  experience  can  only  be  gained  through  the  conduct  of  actual  operations. 
Soldiers  in  our  experiment  spent  only  between  1.5  to  2  hours  completing  the  experimental  tasks. 
For  half  the  participants,  this  included  the  TDST.  Analysis  of  Soldier  responses  to  the  question 
about  previous  training  indicated  the  difficult  nature  of  training  threat  detection  skills.  Though 
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the  skills  are  paramount  to  obtain,  especially  for  the  operational  environments  Soldiers  find 
themselves  in  training  such  skills  and  perhaps,  even  maintaining  them  requires  consistent 
application  and  practices. 


Discussion 

The  quantitative  findings  across  experimental  tasks  indicate  that  training  was  no  more 
effective  at  improving  performance  than  was  the  practice  that  resulted  from  completing  the  pre¬ 
test  and  post-test.  Practice  effects  appeared  to  be  consistent  and  robust.  In  each  task,  Soldiers’ 
performance  improved  over  time,  and  the  effect  sizes  were  relatively  large  in  some  cases  (e.g., 
partial  eta '  >  .30).  Thus,  to  the  extent  that  the  TDST  promotes  practicing  the  skills  required  for 
detecting  threats,  it  improves  performance.  However,  to  be  more  certain  of  this  potential 
improvement,  future  research  would  have  to  tease  apart  the  potential  impact  of  different 
experimental  designs  as  well  as  the  influence  of  different  measures  of  performance. 

A  longitudinal  research  program  examining  the  longer-term  effects  of  completing  the 
trainer  could  be  worth  pursuing.  In  the  current  research,  the  results  showed  an  improvement 
from  pre-  to  post-test  on  almost  all  dependent  variables  of  interest.  This  indicates  that  we 
accurately  identified  the  skills  requisite  for  threat  detection  (Vowels,  2010;  Zimmerman,  et  ah, 
2011;  Zimmerman,  et  ah,  in  preparation;  Zimmerman,  et  ah,  2012).  If  testing  can  demonstrate 
an  improvement  in  the  skills  necessary  for  threat  detection  in  two  hours  or  less  using  an  acute, 
cognitive  training  program,  it  is  worth  investigating  the  effects  that  might  occur  over  several 
weeks  to  months  of  training.  A  potential  methodology  that  has  demonstrated  recent  success  is 
retrieval-based  or  test-enhanced  learning.  Results  from  previous  research  have  consistently 
shown  the  enhancement  of  short-term  retention  and  long-term  retention  of  material  (Roediger  & 
Butler,  2011;  Roediger,  Agarwal,  McDaniel,  &  McDermott,  2011).  Though  typically  examined 
using  grade  school  populations,  it  is  possible  to  expand  the  procedure  to  a  military  population. 
Such  testing  could  provide  a  legitimate  examination  of  training  to  enhance  retrieval-based 
memory  and  recognitional  decision-making  in  an  applied  setting  (U.S.  military).  See  also 
Kornell  and  Rhodes  (2013)  concerning  the  influence  of  feedback  in  test-enhanced  learning. 

In  the  present  research,  qualitative  analysis  focused  on  contextual  differences  in  Soldiers 
responses.  Practice  effects  were  noted  during  some  of  the  analyses  as  the  details  provided  during 
the  post-test  tasks  tended  to  decrease  relative  to  pre-test  tasks.  In  addition,  analysis  occasionally 
revealed  differences  in  the  training  groups,  but  in  the  opposite  direction  than  anticipated. 

Though  assigned  randomly  to  conditions,  Soldiers  in  the  no-training  group,  relative  to  the 
training  group,  tended  to  provide  explanations  with  more  details  and  offered  a  greater  number  of 
strategies  for  overcoming  threats.  This  occurred  across  the  pre-  and  post-tests.  Overall,  any 
differences  in  answer  content  were  minor  across  the  conditions.  This  could  be  due  to  the  nature 
of  the  tasks  or  to  fatigue.  Requiring  Soldiers  to  write  answers  on  multiple  trials,  and  across 
similar  tasks,  might  have  reduced  their  motivation  to  respond  at  length. 

Limitations  of  this  experiment  were  the  predicted  small  effect  of  training  and  a  small 
sample  size.  A  larger  sample  size  would  increase  the  likelihood  of  observing  small  effects. 
However,  it  is  important  to  have  realistic  expectations  about  the  magnitude  of  the  impact  of  the 
trainer  on  performance.  Given  the  present  results,  we  have  the  benefit  of  not  overestimating  how 
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impactful  the  trainer  might  be.  Another  limitation  in  this  experiment  is  that  the  experimental  test 
tasks  were  similar  to  the  tasks  used  in  the  trainer,  thus,  it  is  not  surprising  that  practice  effects 
seemed  to  “wash  out”  any  effects  of  training.  Perhaps  a  third  comparison  group  of  Soldiers  who 
had  less  opportunity  for  practice  would  have  benefited  this  research.  For  practical  purposes, 
however,  this  third  group  was  not  feasible  in  the  present  experiment.  Another  solution  for  future 
research  is  to  create  tasks  that  do  not  resemble  the  tasks  in  the  trainer,  but  provide  the  same 
measures  required  for  the  hypotheses. 

Future  research  should  tease  apart  the  effects  of  practice  and  the  effects  of  training  via  the 
TDST  on  threat  detection  perfonnance.  For  example,  protocols  could  incorporate  differing 
amounts  of  practice  or  multiple  training  sessions  over  time.  Future  experiments  should  test 
threat  detection  in  real-world  scenarios.  Placing  participants  in  live-action  scenarios  might 
mitigate  practice  as  a  confound  because  the  critical  tasks  will  likely  be  significantly  different 
from  those  experienced  in  the  trainer.  This  would  remove  the  writing  requirement  over  multiple 
trials  and  may  reduce  the  effects  of  fatigue.  Finally,  live  scenarios  provide  the  opportunity  to 
examine  the  effect  of  the  trainer  in  a  more  ecologically  valid  context.  Future  research  might  also 
benefit  from  testing  Soldiers  with  different  levels  of  training  and  experience.  The  present 
experiment  tested  Soldiers  with  very  little  training  and  experience.  It  may  be  interesting  to 
explore  whether  more  or  less  trained  Soldiers  benefit  to  differing  degrees  from  completing  the 
TDST. 


The  Army  continues  to  operate  in  asymmetric  warfare  environments,  which  are  marked 
by  high  uncertainty,  time  pressure,  and  high-stakes  decisions.  Hybrid  strategies  and  tactics, 
involving  a  mix  of  standard  and  irregular  warfare  methods  and  technologies,  are  assumed  to 
influence  operations  for  the  foreseeable  future  (TRADOC  Pamphlet  525-3-0,  The  U.S.  Army 
Capstone  Concept).  Given  that,  even  organizations  that  largely  field  technologies  for  the 
irregular  battlefield,  also  understand  the  best  weapon  to  defeat  asymmetric  threats  is  a  well 
educated  and  well  trained  Army  Soldier  (Bosker,  2012).  As  such,  the  TDST  acts  as  a 
supplemental  resource  that  directly  provides  needed  training  and  practice  for  the  Anny’s  most 
crucial  asymmetric  deterrent,  the  human  Soldier. 
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Appendix  A 


Situation  Assessment  Questions 

Following  the  drag-and-drop  exercise,  participants  will  answer  the  following  questions: 

1 .  Describe  what  is  happening  in  the  scene. 

2.  Which  element(s)  in  the  scene  is  the  biggest  threat? 

3.  What  makes  you  believe  that  these  elements  are  a  threat? 

4.  What  are  the  challenges  to  threat  detection  in  this  scene  (or  what  makes  threat 
detection  difficult)? 
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Appendix  B 


Decision-Making  Exercises  (DMX) 


After  reading  each  of  the  DMXs,  participants  answered  the  following  questions: 

1 .  Describe  any  potential  threats  present  in  this  scene  and  explain  the  meaning  and 
implications  of  these  threats.  You  have  two  minutes  to  write  a  response. 

2.  List  three  to  five  factors  that  make  this  situation  dangerous  and  provide  a  brief 
explanation  of  why  each  is  dangerous. 

3.  Name  the  one  potential  threat  you  are  most  concerned  about.  What  strategies  would 
you  use  to  detect  and  mitigate  that  threat? 

4.  List  the  top  three  challenges  to  detecting  threats  in  this  situation  and  state  why  they 
are  challenges. 

These  questions  followed  the  last  sentence  in  each  of  the  below  DMXs: 
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DMX  Title:  Road  Construction 


You  are  Bravo  Co  1/5  CAV.  Your  company  mission  is  to  conduct  COIN  operations,  with  the 
frequent  use  of  CERP  funds  to  build  community  infrastructure  and  win  the  hearts  and  minds  of 
the  local  populace.  When  you  first  arrived  at  your  AOR,  there  were  very  few  incidents  and  the 
community  generally  supported  a  U.S.  military  presence.  In  recent  months,  sentiment  has 
changed.  There  have  been  multiple  IED  incidents  in  the  past  two  weeks  along  route  Grover,  the 
major  supply  route  for  U.S.  forces  in  your  sector,  with  several  small-arms  attacks  and  attempted 
ambushes  on  supply  convoys  after  IEDs  pinned  them  in  place.  This  has  resulted  in  increased 
security  patrols  along  with  a  big  push  to  engage  with  community  members,  in  part  to  win  back 
their  approval,  and  in  part  to  detennine  why  the  U.S.  is  losing  support. 

For  this  reason,  your  unit  goes  out  on  frequent  vehicle  and  foot  patrols  with  the  goal  of 
interacting  with  the  village  locals.  On  this  afternoon,  your  convoy  is  driving  into  the  village  so 
the  platoon  leader  can  do  a  KLE  with  the  village  elder.  As  your  unit  conducted  movement  along 
route  Grover,  you  came  upon  several  individuals  doing  work  along  the  road.  You  do  not  have 
any  record  of  any  scheduled  work  in  this  area,  but  with  the  recent  upswing  in  new  CERP  efforts, 
this  might  be  legitimate  work.  You  are  stopped  on  the  road  while  the  squad  leader  attempts  to 
determine  if  any  work  is  scheduled  for  this  area. 

This  photo  shows  what  you  are  viewing  at  the  point  where  your  unit  has  stopped. 
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Title:  Iraq  Elections 


You  have  temporarily  been  assigned  to  the  2nd  Recon  Division,  U.S.  Marine  Corps  for  upcoming 
security  at  various  checkpoints  throughout  the  city  of  Baghdad,  Iraq.  The  government  of  Iraq 
designated  these  checkpoints  as  election  booths.  The  military  put  added  security  measures  into 
effect  because  terrorists  have  publicly  vowed  to  increase  violent  activities  as  Election  Day 
approaches.  The  threat  of  sectarian  violence  also  increases  as  one  or  both  of  the  parties  on  the 
ballot  sway  favor. 

Your  ROE  are  to  stop  and  screen  all  traffic  between  election  hours  up  through  the  date  of 
election.  The  authorization  of  deadly  force  is  approved  when  peaceful  options  have  been 
exhausted  and  or  for  immediate  protection  of  the  security  forces.  You  are  allowed  to  apprehend 
any  or  all  personnel  that  are  acting  suspiciously,  appear  threatening,  or  are  perpetrating  violence 
against  other  citizens  of  Iraq. 

It  is  currently  late  morning  and  your  unit  is  doing  initial  security  checks  of  each  checkpoint  in 
your  AO.  You  are  approaching  the  third  checkpoint  of  the  day.  The  other  two  checkpoints  were 
secure  and  operations  to  set  up  election  booths  were  progressing  without  incident.  The  personnel 
operating  the  checkpoints  were  known  members  of  Iraqi  security  forces  trained  by  U.S.  Soldiers. 
As  your  unit  approaches  the  third  checkpoint,  there  appears  to  be  no  evidence  of  the  election 
booth  supplies  and,  unlike  the  first  two  checkpoints,  no  activities  are  underway  to  set  up  the 
election  booths.  You  have  not  yet  identified  the  checkpoint  personnel. 
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This  photo  shows  your  current  view  of  the  checkpoint  as  your  unit  approaches. 
Title:  ANP  Incident 


You  are  at  a  joint  compound,  manning  a  post  with  a  member  of  the  Afghan  National  Police 
(ANP).  You  have  worked  with  this  the  ANP  member  several  times  before.  On  this  occasion,  the 
ANP  member  is  acting  erratically  and  arguing  with  everyone.  He  was  repeatedly  asking  you  to 
exchange  some  U.S.  dollars  for  rupees.  You  kept  refusing  and  finally  asked  the  ANP  member 
stop  asking  you.  The  ANP  member  became  angry  and  began  acting  carelessly  with  his  weapon, 
pointing  it  in  the  sky  and  walking  away  from  the  post.  You  finally  called  the  ANP  member’s 
sergeant  to  report  the  ANP  member’s  behavior.  The  ANP  sergeant  ordered  the  ANP  member  to 
return  to  his  post.  The  ANP  member  returned  and  began  threatening  to  call  the  Taliban.  You  are 
now  at  your  post  watching  as  the  ANP  member’s  erratic  actions  continue. 
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Title:  Convoy  Operation 


You  are  the  gunner  in  your  truck  and  the  lowest-ranking  member  of  the  patrol.  Your  mounted 
patrol  takes  a  break  at  a  location  that  you  consider  relatively  safe.  You  are  in  the  turret  on  watch 
during  the  break.  The  other  vehicle  occupants  are  outside  the  vehicle.  Your  vehicle  is 
positioned  best  to  cover  a  slight  draw  approaching  your  position. 

A  local  national  male  of  military  age  appears  in  the  draw  and  rapidly  approaches  your  position. 
You  begin  using  EOF  procedures  -  yelling  at  him  to  stop,  waving  at  him  to  stop,  and  then 
pointing  your  weapon  toward  him.  You  are  unsure  where  the  rest  of  your  patrol  is  while  you 
continue  to  monitor  the  man,  though  from  your  vantage  point  you  know  they  are  not  in  your  field 
of  fire  and  they  are  nearby.  The  man  continues  to  approach  at  a  high  rate  of  speed  and  closes  to 
within  20  meters.  You  cannot  tell  what  his  intentions  are.  You  shout  for  your  team  leader,  but 
he  does  not  hear  you. 

This  photo  shows  what  you  are  viewing  at  the  point  where  your  unit  has  stopped. 
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Appendix  C 


Demographic  Questions 


1.  Time  in  service  (years) 

2.  Current  rank 

3.  Time  in  current  rank  (months) 

4.  Current  MOS 

5.  Age 

6.  Have  you  ever  deployed?  Yes  No 

7.  If  yes,  how  many  times  have  you  deployed? 

8.  Location  of  most  recent  deployment  (city  or  cities  and  country) 

9.  MOS  while  on  your  most  recent  deployment: 

10.  How  often  did  you  go  outside  the  wire  on  your  most  recent  deployment? 

Never  Less  than  once  a  month  Once  a  month  More  than  once  a  month 

Once  a  week  More  than  once  a  week  Everyday 

1 1 .  Describe  some  of  your  duties  during  your  most  recent  deployment  and  indicate  if  you 
conducted  dismounted  patrols  (present  patrols  and/or  clearing  operations)  and  route 
clearance: 

12.  Describe  any  training  you  have  received  that  improved  your  ability  to  detect  threats  and 
indicate  approximate  date  of  training  month/year: 
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